Android-ElectricSheep-Automator
view release on metacpan or search on metacpan
Makefile.PL view on Meta::CPAN
use ExtUtils::MakeMaker;
sub MY::libscan {
my( $mm, $file ) = @_;
return if $file =~ /^push_to_GIT$/; # SKIP the git dir
return if $file =~ /^tmp$/; # SKIP the git dir
return if $file =~ /^bin$/; # SKIP the git dir
return if $file =~ /^experiments$/; # private folder
return if $file =~ /^doc$/; # private folder
return if $file =~ /^forums$/; # private folder
return if $file =~ /\.lock.*$/; # SKIP editor files
return $file;
}
my %WriteMakefileArgs = (
NAME => 'Android::ElectricSheep::Automator',
AUTHOR => q{Andreas Hadjiprocopis <bliako@cpan.org>},
VERSION_FROM => 'lib/Android/ElectricSheep/Automator.pm',
ABSTRACT_FROM => 'lib/Android/ElectricSheep/Automator.pm',
LICENSE => 'artistic_2',
MIN_PERL_VERSION => '5.006',
widget's location in order to get the focus. And then it enters the
text. You need to find the position of the desired text-input widget
by first getting the current screen UI (using dump_current_screen_ui)
and then using an XPath selector to identify the desired widget by
name/id/attributes. See the source code of method send_message() in
file lib/Android/ElectricSheep/Automator/Plugins/Apps/Viber.pm for
how this is done for the message-sending text-input widget of the
Viber app.
$params is a HASH_REF which must contain text and one of the two
position (of the text-edit widget) specifiers position or bounds:
text
the text to write on the text edit widget. At the moment, this must
be plain ASCII string, not unicode. No spaces are accepted. Each
space character must be replaced with %s.
position
should be an ARRAY_REF as the X,Y coordinates of the point to "tap"
in order to get the focus of the text edit widget, preceding the
text input.
bounds
should be an ARRAY_REF of a bounding rectangle of the widget to
tap, in order to get the focus, preceding the text input. Which
contains two ARRAY_REFs for the top-left and bottom-right
coordinates, e.g. [ [tlX,tlY], [brX,brY] ] . This is convenient
when the widget is extracted from an XML dump of the UI (see
"dump_current_screen_ui()") which contains exactly this bounding
clear_input_field($params)
It clears the contents of a text-input widget at specified location.
There are several ways to do this. The simplest way (with
keycombination) does not work in some devices, in which case a
failsafe way is employed which deletes characters one after the other
for 250 times.
$params is a HASH_REF which must contain one of the two position (of
the text-edit widget) specifiers position or bounds:
position
should be an ARRAY_REF as the X,Y coordinates of the point to "tap"
in order to get the focus of the text edit widget, preceding the
text input.
bounds
should be an ARRAY_REF of a bounding rectangle of the widget to
tap, in order to get the focus, preceding the text input. Which
contains two ARRAY_REFs for the top-left and bottom-right
coordinates, e.g. [ [tlX,tlY], [brX,brY] ] . This is convenient
when the widget is extracted from an XML dump of the UI (see
"dump_current_screen_ui()") which contains exactly this bounding
rectangle.
num-characters
how many times to press the backspace? Default is 250! But if you
know the length of the text currently at the text-edit widget then
enter this here.
It returns 0 on success, 1 on failure.
home_screen()
Go to the "home" screen.
It returns 0 on success, 1 on failure.
and running with, for example, emulator -avd Pixel_2_API_30_x86_ . See
section "Android Emulators" for how to install, list and run them
buggers.
Testing will not send any messages via the device's apps. E.g. the
plugin Android::ElectricSheep::Automator::Plugins::Apps::Viber will not
send a message via Viber but it will mock it.
The live tests will sometimes fail because, so far, something
unexpected happened in the device. For example, in testing sending
input text to a text-edit widget, the calendar will be opened and a new
entry will be added and its text-edit widget will be targeted. Well,
sometimes the calendar app will give you some notification on startup
and this messes up with the focus. Other times, the OS will detect that
some app is taking too long to launch and pops up a notification about
"something is not responding, shall I close it". This steals the focus
and sometimes it causes the tests to fail.
PREREQUISITES
Android Studio
the text. You need to find the position of the desired
text-input widget by first getting the current screen UI
(using [dump\_current\_screen\_ui](https://metacpan.org/pod/dump_current_screen_ui)) and then using an XPath
selector to identify the desired widget by name/id/attributes.
See the source code of method `send_message()` in file
`lib/Android/ElectricSheep/Automator/Plugins/Apps/Viber.pm`
for how this is done for the message-sending text-input widget
of the Viber app.
`$params` is a HASH\_REF which must contain `text`
and one of the two position (of the text-edit widget)
specifiers `position` or `bounds`:
- **`text`**
the text to write on the text edit widget. At the
moment, this must be plain ASCII string, not unicode.
No spaces are accepted.
Each space character must be replaced with `%s`.
- **`position`**
should be an ARRAY\_REF
as the `X,Y` coordinates of the point to "tap" in order
to get the focus of the text edit widget, preceding the
text input.
- **`bounds`**
should be an ARRAY\_REF of a bounding rectangle
of the widget to tap, in order to get the focus, preceding
the text input. Which contains two ARRAY\_REFs
for the top-left and bottom-right coordinates, e.g.
` [ [tlX,tlY], [brX,brY] ] `. This is convenient
when the widget is extracted from an XML dump of
It clears the contents of a text-input widget
at specified location.
There are several ways to do this. The simplest way
(with `keycombination`) does not work in some
devices, in which case a failsafe way is employed
which deletes characters one after the other for
250 times.
`$params` is a HASH\_REF which must contain
one of the two position (of the text-edit widget)
specifiers `position` or `bounds`:
- **`position`**
should be an ARRAY\_REF
as the `X,Y` coordinates of the point to "tap" in order
to get the focus of the text edit widget, preceding the
text input.
- **`bounds`**
should be an ARRAY\_REF of a bounding rectangle
of the widget to tap, in order to get the focus, preceding
the text input. Which contains two ARRAY\_REFs
for the top-left and bottom-right coordinates, e.g.
` [ [tlX,tlY], [brX,brY] ] `. This is convenient
when the widget is extracted from an XML dump of
the UI (see ["dump\_current\_screen\_ui()"](#dump_current_screen_ui)) which
contains exactly this bounding rectangle.
- **`num-characters`**
how many times to press the backspace? Default is 250!
But if you know the length of the text currently at
the text-edit widget then enter this here.
It returns `0` on success, `1` on failure.
- home\_screen()
Go to the "home" screen.
It returns `0` on success, `1` on failure.
- wake\_up()
`emulator -avd Pixel_2_API_30_x86_` . See section
["Android Emulators"](#android-emulators) for how to install, list and run them
buggers.
Testing will not send any messages via the device's apps.
E.g. the plugin [Android::ElectricSheep::Automator::Plugins::Apps::Viber](https://metacpan.org/pod/Android%3A%3AElectricSheep%3A%3AAutomator%3A%3APlugins%3A%3AApps%3A%3AViber)
will not send a message via Viber but it will mock it.
The live tests will sometimes fail because, so far,
something unexpected happened in the device. For example,
in testing sending input text to a text-edit widget,
the calendar will be opened and a new entry will be added
and its text-edit widget will be targeted. Well, sometimes
the calendar app will give you some notification
on startup and this messes up with the focus.
Other times, the OS will detect that some app is taking too
long to launch and pops up a notification about
"_something is not responding, shall I close it_".
This steals the focus and sometimes it causes
the tests to fail.
# PREREQUISITES
lib/Android/ElectricSheep/Automator.pm view on Meta::CPAN
use Config::JSON::Enhanced;
# it requires v0.002 (which is with my modifications)
#use Android::ADB;
# issue filed for Android::ADB :
# https://rt.cpan.org/Public/Bug/Display.html?id=163391
# until this is resolved I am copying Android::ADB
# into my distribution, fixing the issues and renaming it to
# Android::ElectricSheep::Automator::ADB
# and using that. When the issue is resolved I will go back
# using Android::ADB
# Credits for Android::ADB (now Android::ElectricSheep::Automator::ADB)
# go to Marius Gavrilescu (marius@ieval.ro)
# as seen in:
# https://metacpan.org/pod/Android::ADB
#
use Android::ElectricSheep::Automator::ADB;
use File::Temp qw/tempfile/;
use Cwd;
use FindBin;
use Time::HiRes qw/usleep/;
use Image::PNG;
lib/Android/ElectricSheep/Automator.pm view on Meta::CPAN
@position = ( int(($m->[1]->[0] + $m->[0]->[0])/2), int(($m->[1]->[1] + $m->[0]->[1])/2) );
} else { $log->error("${whoami} (via $parent), line ".__LINE__." : error, input parameter 'position' (as ['x','y']) or 'bounds' (as [lefttopX,lefttopY],[bottomrightX,bottomrighY]) was not specified."); return 1 }
# optional text, else we send just '' (but we clicked on it)
my $text = (exists($params->{'text'}) && defined($params->{'text'})) ? $params->{'text'} : '';
# sanitise the text a bit
# replace spaces with %s,
# also newlines seem not to be supported so replaces these as well
$text =~ s/[\n \t]/%s/g;
# first tap on the text edit widget at the specified coordinates to get focus
if( $self->tap($params) ){ $log->error("${whoami} (via $parent), line ".__LINE__." : error, failed to tap on the position of the recipient of the text input"); return 1 }
usleep(0.8);
# and send the text
# adb shell input text 'hello%sworld'
# does not support unicode and also spaces must be converted to %s (already have done this)
my @cmd = ('input', 'text', $text);
if( $verbosity > 0 ){ $log->info("${whoami} (via $parent), line ".__LINE__." : sending command to adb: @cmd") }
my $res = $self->adb->shell(@cmd);
if( ! defined $res ){ $log->error(join(" ", @cmd)."\n${whoami} (via $parent), line ".__LINE__." : error, above shell command has failed, got undefined result, most likely shell command did not run at all, this should not be happening."); return 1 }
lib/Android/ElectricSheep/Automator.pm view on Meta::CPAN
if( ! $self->is_device_connected() ){ $log->error("${whoami} (via $parent), line ".__LINE__." : error, you need to connect a device to the desktop and ALSO explicitly call ".'connect_device()'." before calling this."); return undef }
my (@position, $m);
if( exists($params->{'position'}) && defined($m=$params->{'position'}) ){
@position = ($m->[0], $m->[1]);
} elsif( exists($params->{'bounds'}) && defined($m=$params->{'bounds'}) ){
@position = ( int(($m->[1]->[0] + $m->[0]->[0])/2), int(($m->[1]->[1] + $m->[0]->[1])/2) );
} else { $log->error("${whoami} (via $parent), line ".__LINE__." : error, input parameter 'position' (as ['x','y']) or 'bounds' (as [lefttopX,lefttopY],[bottomrightX,bottomrighY]) was not specified."); return 1 }
# first tap on the text edit widget at the specified coordinates to get focus
if( $self->tap($params) ){ $log->error("${whoami} (via $parent), line ".__LINE__." : error, failed to tap on the position of the recipient of the text input"); return 1 }
usleep(0.8);
# from: https://stackoverflow.com/questions/32433303/clear-edit-text-adb
# the simplest way is input keycombination 113 29 && input keyevent 67
# but may not work
# then we try the lame way by erasing all chars one after the other
# part1:
my @cmd = ('input', 'keycombination', '113', '29');
if( $verbosity > 0 ){ $log->info("${whoami} (via $parent), line ".__LINE__." : sending command to adb: @cmd") }
my $res = $self->adb->shell(@cmd);
if( ! defined($res) || ($res->[0] != 0) || ($res->[1]=~/Error: Unknown command/) || ($res->[2]=~/Error: Unknown command/) ){
$log->warn(join(" ", @cmd)."\n${whoami} (via $parent), line ".__LINE__." : error, above shell command has failed which happens and now will try an alternative ...");
# alternative/part1
@cmd = ('input', 'keyevent', 'KEYCODE_MOVE_END');
if( $verbosity > 0 ){ $log->info("${whoami} (via $parent), line ".__LINE__." : sending command to adb: @cmd") }
$res = $self->adb->shell(@cmd);
if( ! defined $res ){ $log->error(join(" ", @cmd)."\n${whoami} (via $parent), line ".__LINE__." : error, above shell command has failed, got undefined result, most likely shell command did not run at all, this should not be happening. Info: this is...
if( $res->[0] != 0 ){ $log->error(join(" ", @cmd)."\n${whoami} (via $parent), line ".__LINE__." : error, above shell command has failed (Info: this is alternative part1), with:\nSTDOUT:\n".$res->[1]."\n\nSTDERR:\n".$res->[2]."\nEND."); return 1 }
# alternative/part2
# optional number of chars in the text-edit box, meaning how many
# times to press backspace, default is here (250)
# this is only needed for the second method (the failsafe)
my $numchars = (exists($params->{'num-characters'}) && defined($params->{'num-characters'}) && ($params->{'num-characters'}=~/^\d+$/) ) ? $params->{'num-characters'} : 250;
@cmd = ('input', 'keyevent', '--longpress', ('KEYCODE_DEL')x$numchars);
if( $verbosity > 0 ){ $log->info("${whoami} (via $parent), line ".__LINE__." : sending command to adb: @cmd") }
$res = $self->adb->shell(@cmd);
if( ! defined $res ){ $log->error(join(" ", @cmd)."\n${whoami} (via $parent), line ".__LINE__." : error, above shell command has failed, got undefined result, most likely shell command did not run at all, this should not be happening. Info: this is...
if( $res->[0] != 0 ){ $log->error(join(" ", @cmd)."\n${whoami} (via $parent), line ".__LINE__." : error, above shell command has failed (Info: this is alternative part1), with:\nSTDOUT:\n".$res->[1]."\n\nSTDERR:\n".$res->[2]."\nEND."); return 1 }
} else {
# part2
lib/Android/ElectricSheep/Automator.pm view on Meta::CPAN
the text. You need to find the position of the desired
text-input widget by first getting the current screen UI
(using L<dump_current_screen_ui>) and then using an XPath
selector to identify the desired widget by name/id/attributes.
See the source code of method C<send_message()> in file
C<lib/Android/ElectricSheep/Automator/Plugins/Apps/Viber.pm>
for how this is done for the message-sending text-input widget
of the Viber app.
C<$params> is a HASH_REF which must contain C<text>
and one of the two position (of the text-edit widget)
specifiers C<position> or C<bounds>:
=over 4
=item B<C<text>>
the text to write on the text edit widget. At the
moment, this must be plain ASCII string, not unicode.
No spaces are accepted.
Each space character must be replaced with C<%s>.
=item B<C<position>>
should be an ARRAY_REF
as the C<X,Y> coordinates of the point to "tap" in order
to get the focus of the text edit widget, preceding the
text input.
=item B<C<bounds>>
should be an ARRAY_REF of a bounding rectangle
of the widget to tap, in order to get the focus, preceding
the text input. Which contains two ARRAY_REFs
for the top-left and bottom-right coordinates, e.g.
C< [ [tlX,tlY], [brX,brY] ] >. This is convenient
when the widget is extracted from an XML dump of
lib/Android/ElectricSheep/Automator.pm view on Meta::CPAN
It clears the contents of a text-input widget
at specified location.
There are several ways to do this. The simplest way
(with C<keycombination>) does not work in some
devices, in which case a failsafe way is employed
which deletes characters one after the other for
250 times.
C<$params> is a HASH_REF which must contain
one of the two position (of the text-edit widget)
specifiers C<position> or C<bounds>:
=over 4
=item B<C<position>>
should be an ARRAY_REF
as the C<X,Y> coordinates of the point to "tap" in order
to get the focus of the text edit widget, preceding the
text input.
=item B<C<bounds>>
should be an ARRAY_REF of a bounding rectangle
of the widget to tap, in order to get the focus, preceding
the text input. Which contains two ARRAY_REFs
for the top-left and bottom-right coordinates, e.g.
C< [ [tlX,tlY], [brX,brY] ] >. This is convenient
when the widget is extracted from an XML dump of
the UI (see L</dump_current_screen_ui()>) which
contains exactly this bounding rectangle.
=item B<C<num-characters>>
how many times to press the backspace? Default is 250!
But if you know the length of the text currently at
the text-edit widget then enter this here.
=back
It returns C<0> on success, C<1> on failure.
=item home_screen()
Go to the "home" screen.
It returns C<0> on success, C<1> on failure.
lib/Android/ElectricSheep/Automator.pm view on Meta::CPAN
C<emulator -avd Pixel_2_API_30_x86_> . See section
L<Android Emulators> for how to install, list and run them
buggers.
Testing will not send any messages via the device's apps.
E.g. the plugin L<Android::ElectricSheep::Automator::Plugins::Apps::Viber>
will not send a message via Viber but it will mock it.
The live tests will sometimes fail because, so far,
something unexpected happened in the device. For example,
in testing sending input text to a text-edit widget,
the calendar will be opened and a new entry will be added
and its text-edit widget will be targeted. Well, sometimes
the calendar app will give you some notification
on startup and this messes up with the focus.
Other times, the OS will detect that some app is taking too
long to launch and pops up a notification about
"I<something is not responding, shall I close it>".
This steals the focus and sometimes it causes
the tests to fail.
=head1 PREREQUISITES
lib/Android/ElectricSheep/Automator/Plugins/Apps/Viber.pm view on Meta::CPAN
# click the Recipient contact name at the bottom
$boundstr = $node->getAttribute('bounds');
if( ! defined $boundstr ){ $log->error("${node}\n\n${whoami} (via $parent), line ".__LINE__." : error, above node does not have attribute 'bounds'."); return undef }
if( $boundstr !~ /\[\s*(\d+)\s*,\s*(\d+)\]\s*\[\s*(\d+)\s*,\s*(\d+)\]/ ){ $log->error("${whoami} (via $parent), line ".__LINE__." : error, failed to parse bounds string: '$boundstr'."); return undef }
$bounds = [[$1,$2],[$3,$4]];
# click it
if( $self->mother->tap({'bounds' => $bounds}) ){ $log->error("${whoami} (via $parent), line ".__LINE__." : error, failed to tap on $bounds."); return undef }
usleep(1.5);
# Put the text into the text-edit Message... (note: ... is unicode ellipses something)
# get the UI
$outfile = defined($outbase) ? $outbase.'_chat.xml' : undef;
$ui = $self->mother->dump_current_screen_ui({'filename'=>$outfile});
$dom = $ui->{'XML::LibXML'};
$xc = $ui->{'XML::LibXML::XPathContext'};
$asel = '//node'
. '['
. ' matches(@class,\'EditText$\',"i")'
. ' and @resource-id="com.viber.voip:id/send_text"'
. ' and @package="com.viber.voip"'
xt/live/400-input_text-clear_input_field.t view on Meta::CPAN
# click the add-button found
my $boundstr = $addbutton->getAttribute('bounds');
ok(defined($boundstr), "bounds of the widget found: ${boundstr}") or BAIL_OUT;
ok($boundstr =~ /\[\s*(\d+)\s*,\s*(\d+)\]\s*\[\s*(\d+)\s*,\s*(\d+)\]/, "bounds of the widget parsed: ${boundstr}") or BAIL_OUT;
my $bounds = [[$1,$2],[$3,$4]];
is($mother->tap({'bounds'=>$bounds}), 0, 'add (+) button of the app has been clicked.') or BAIL_OUT(perl2dump($bounds)."no, failed to tap the add button with above bounds");
sleep(1);
# and now we have a text-edit widget
# dump the ui
$ui = $mother->dump_current_screen_ui();
ok(defined($ui), 'dump_current_screen_ui()'." : got UI dump of the running app's screen.") or BAIL_OUT;
$dom = $ui->{'XML::LibXML'};
$xc = $ui->{'XML::LibXML::XPathContext'};
$asel =
'//node[@resource-id="com.google.android.deskclock:id/open_search_view_edit_text"'
.' and matches(@class, \'Edit\',"i")'
.' and matches(@text, \'city\',"i")'
.']'
;
@nodes = $xc->findnodes($asel);
#for my $anode (@nodes){ print $anode; } die 123;
$N = scalar @nodes;
is($N, 1, "app clock : found text-edit widget from UI of the app with this selector: ${asel}") or BAIL_OUT(${ui}->{'raw'}."\nno it failed with above XML dump");
my $texteditbox = $nodes[0];
# click the add-button found
$boundstr = $texteditbox->getAttribute('bounds');
ok(defined($boundstr), "bounds of the widget found: ${boundstr}") or BAIL_OUT;
ok($boundstr =~ /\[\s*(\d+)\s*,\s*(\d+)\]\s*\[\s*(\d+)\s*,\s*(\d+)\]/, "bounds of the widget parsed: ${boundstr}") or BAIL_OUT;
$bounds = [ [$1,$2], [$3,$4] ];
my $text = 'hello%sworld';
is($mother->input_text({'bounds'=>$bounds, 'text'=>$text}), 0, "text ($text) has been entered in the text-edit widget.") or BAIL_OUT(perl2dump($bounds)."no, failed to input text ($text) in the text-edit widget with above bounds");
# now clear the text
is($mother->clear_input_field({
'bounds'=>$bounds,
'num-characters' => 15
}), 0, 'clear_input_field()'." : the text entered in the text field earlier must now be cleared.") or BAIL_OUT;
# close apps
$res = $mother->close_app({
'package' => $package
( run in 0.790 second using v1.01-cache-2.11-cpan-de7293f3b23 )