Android-ElectricSheep-Automator
view release on metacpan or search on metacpan
NAME
Android::ElectricSheep::Automator - Do Androids Dream of Electric
Sheep? Smartphone control from your desktop.
VERSION
Version 0.09
WARNING
Current distribution is extremely alpha. API may change.
SYNOPSIS
The present package fascilitates the control of a USB-debugging-enabled
Android device, e.g. a real smartphone, or an emulated (virtual)
Android device, from your desktop computer using Perl. It's basically a
thickishly-thin wrapper to the omnipotent Android Debug Bridge (adb)
program.
Note that absolutely nothing is installed on the connected device,
neither any of its settings will be modified by this package. See "WILL
ANYTHING BE INSTALLED ON THE DEVICE?".
use Android::ElectricSheep::Automator;
my $mother = Android::ElectricSheep::Automator->new({
# optional as there is a default, but you may have
# problems with the location of the adb executable
'configfile' => $configfile,
'verbosity' => 1,
# we already have a device connected and ready to control
'device-is-connected' => 1,
});
# find the devices connected to desktop and set one.
my @devices = $mother->adb->devices;
$mother->connect_device({'serial' => $devices->[0]->serial})
or die;
# no device identification is required for the method call
# if there is only one connected device:
$mother->connect_device() if scalar(@devices)==0;
# Go Home
$mother->home_screen() or die;
# swipe up/down/left/right
$mother->swipe({'direction'=>up}) or die;
# dt is the time to swipe in millis,
# the shorter the faster the swipe
$mother->swipe({'direction'=>left, 'dt'=>100}) or die;
# tap
$mother->tap({'position'=>[100,200]});
# uses swipe() to move in screens (horizontally):
$mother->next_screen() or die;
$mother->previous_screen() or die;
# bottom navigation:
# the "triangle" back button
$mother->navigation_menu_back_button() or die;
# the "circle" home button
$mother->navigation_menu_home_button() or die;
# the "square" overview button
$mother->navigation_menu_overview_button() or die;
# open/close apps
$mother->open_app({'package'=>qr/calendar$/i}) or die;
$mother->close_app({'package'=>qr/calendar$/i}) or die;
# push pull files
$mother->adb->pull($deviceFile, $localFile);
$mother->adb->push($localFile, $deviceFileOrDir);
# guess what!
my $xmlstr = $mother->dump_current_screen_ui();
# Pull the apk(s) for an app from device and save locally
my $res = $mother->pull_app_apk_from_device({
package => 'com.google.android.calendar'
# or qr/calendar/i
'output-dir' => '/tmp/apks-of-calendar-app',
});
print $res->{'com.google.android.calendar'}->[0]->['local-path'};
# Install apk(s) for an app onto the device
$mother->install_app({
'apk-filename' => ['/tmp/apks/base.apk', '/tmp/apks/config.apk'],
# or just a string scalar '/tmp/apks/1.apk'
# optional params to the adb install command
'install-parameters' => ['-r', '-g']
});
CONSTRUCTOR
new($params)
Creates a new Android::ElectricSheep::Automator object. $params is a
hash reference used to pass initialization options which may or should
include the following:
* confighash or configfile
the configuration file holds configuration parameters. Its format is
"enhanced" JSON (see Config::JSON::Enhanced) which is basically JSON
which allows comments between </* and */> .
Here is an example configuration file to get you started:
{
"adb" : {
"path-to-executable" : "/usr/local/android-sdk/platform-tools/adb"
},
"debug" : {
"verbosity" : 0,
</* cleanup temp files on exit */>
"cleanup" : 1
},
"logger" : {
</* log to file if you uncomment this, else console */>
"filename" : "my.log"
}
}
All sections in the configuration are mandatory. Setting "adb" to the
wrong path will yield problems.
confighash is a hash of configuration options with structure as above
and can be supplied to the constructor instead of the configuration
file.
If no configuration is specified, then a default configuration will
be used. In this case please specify adb-path-to-executable to point
to the location of adb. Most likely the default path will not work
for you.
* adb-path-to-executable
optionally specify the path to the adb executable in your desktop
system. This will override the setting 'adb'->'path-to-executable'
in the configuration, if it was provided. Use this option if you are
not providing any configuration and so the default configuration will
be used. But it will most likely fail because of this path not being
correct for your system. So, if you are going to omit providing a
configuration and the default configuration will be used do specify
the adb path via this option (but you don't have to and your mileage
may vary).
* device-serial or device-object
optionally specify the serial of a device to connect to on
instantiation, or a
Android::ElectricSheep::Automator::DeviceProperties object you
already have handy. Alternatively, use "connect_device($params)" to
set the connected device at a later time. Note that there is no need
to specify a device if there is exactly one connected device.
* adb
optionally specify an already created Android::ADB object. Otherwise,
a fresh object will be created based on the configuration under the
adb section of the configuration.
* device-is-connected
optionally set it to 1 in order to communicate with the device and
get some information about it like screen size, resolution,
orientation, etc. And also allow use of functionality which needs
communicating with a device like "swipe($params)",
"home_screen($params)", "open_app($params)", etc. After
instantiation, you can use the method "connect_device($params)" and
"disconnect_device()" for conveying this information to the module.
Also note that if there are more than one devices connected to the
desktop, make sure you specify which one with the device parameter.
Default value is 0.
* logger
optionally specify a logger object to be used (instead of creating a
fresh one). This object must implement these methods: info(), warn(),
error(). Mojo::Log fits perfectly.
* logfile
optionally specify a file to save logging output to. This overrides
the filename key under section logger of the configuration.
* verbosity
optionally specify a verbosity level which will override what the
configuration contains. Default is 0.
* cleanup
optionally specify a flag to clean up any temp files after exit which
will override what the configuration contains. Default is 1, meaning
Yes!.
METHODS
Note:
* ARRAY_REF : my $ar = [1,2,3]; my $ar = \@ahash; my @anarray = @$ar;
* HASH_REF : my $hr = {1=1, 2=>2}; my $hr = \%ahash; my %ahash =
%$hr;>
* In this module parameters to functions are passed as a HASH_REF.
Functions return back objects, ARRAY_REF or HASH_REF.
devices()
Lists all Android devices connected to your desktop and returns these
as an ARRAY_REF which can be empty.
It returns undef on failure.
connect_device($params)
Specifies the current Android device to control. Its use is required
only if you have more than one devices connected. $params is a HASH_REF
which should contain exactly one of the following:
* serial should contain the serial (string) of the connected device
as returned by "devices()".
* device-object should be an already existing
Android::ElectricSheep::Automator::DeviceProperties object.
It returns 0 on success, 1 on failure.
dump_current_screen_ui($params)
It dumps the current screen as XML and returns that as a string,
optionally saving it to the specified file.
$params is a HASH_REF which may or should contain:
* filename
optionally save the returned XML string to the specified file.
It returns undef on failure or the UI XML dump, as a string, on
success.
dump_current_screen_shot($params)
It dumps the current screen as a PNG image and returns that as a
Image::PNG object, optionally saving it to the specified file.
$params is a HASH_REF which may or should contain:
* filename
optionally save the returned XML string to the specified file.
It returns undef on failure or a Image::PNG image, on success.
dump_current_screen_video($params)
It dumps the current screen as MP4 video and saves that in specified
file.
$params is a HASH_REF which may or should contain:
* filename
save the recorded video to the specified file in MP4 format. This is
required.
* time-limit
optionally specify the duration of the recorded video, in seconds.
local-path pointing to the location where the APK was saved, locally.
Note that there is a script available which utilises this method, see
electric-sheep-pull-app-apk.pl.
install_app($params)
It installs the app from its specified APK (bytecode archive and more)
file.
$params is a HASH_REF which should contain:
* apk-filename
The APK filename to install onto the device. It must exist locally,
obviously.
* install-parameters
Optional parameters to be passed on to the adb install command.
Nothing is expected here. Refer to the adb documentation
<https://developer.android.com/tools/adb> for what parameters are
supported. For example, -r is for re-installation of an existing app
and retaining its previous data.
It returns 1 on failure. It returns 0 on success.
Note that there is a script available which utilises this method, see
electric-sheep-install-app.pl.
is_app_running($params)
It checks if the specified app is running on the device. The name of
the app must be exact. Note that you can search for running apps /
commands with extended regular expressions using L/pgrep()>
$params is a HASH_REF which should contain:
* appname
the name of the app to check if it is running. It must be its exact
name. Basically it checks the output of pidof().
It returns undef on failure, 1 if the app is running or 0 if the app is
not running.
find_current_device_properties($params)
It enquires the device currently connected, and specified with
"connect_device($params)", if needed, and returns back an
Android::ElectricSheep::Automator::DeviceProperties object containing
this information, for example screen size, resolution, serial number,
etc.
It returns Android::ElectricSheep::Automator::DeviceProperties object
on success or undef on failure.
connect_device()
It signals to our object that there is now a device connected to the
desktop and its enquiry and subsequent control can commence. If this is
not called and neither device-is-connected => 1 is specified as a
parameter to the constructor, then the functionality will be limited
and access to functions like "swipe($params)", "open_app($params)",
etc. will be blocked until the caller signals that a device is now
connected to the desktop.
Using "connect_device($params)" to specify which device to target in
the case of multiple devices connected to the desktop will also call
this method.
This method will try to enquire the connected device about some of its
properties, like screen size, resolution, orientation, serial number
etc. This information will subsequently be available via
$self->device_properties()>.
It returns 0 on success, 1 on failure.
disconnect_device()
Signals to our object that it should consider that there is currently
no device connected to the desktop (irrespective of that is true or
not) which will block access to "swipe($params)", "open_app($params)",
etc.
device_properties()
It returns the currently connected device properties as a
Android::ElectricSheep::Automator::DeviceProperties object or undef if
there is no connected device. The returned object is constructed during
a call to "find_current_device_properties($params)" which is called via
"connect_device($params)" and will persist for the duration of the
connection. However, after a call to "disconnect_device()" this object
will be discarded and undef will be returned.
swipe($params)
Emulates a "swipe" in four directions. Sets the current Android device
to control. It is only required if you have more than one device
connected. $params is a HASH_REF which may or should contain:
* direction
should be one of
up
down
left
right
* dt
denotes the time taken for the swipe in milliseconds. The smaller its
value the faster the swipe. A value of 100 is fast enough to swipe to
the next screen.
It returns 0 on success, 1 on failure.
tap($params)
Emulates a "tap" at the specified location. $params is a HASH_REF which
must contain one of the following items:
* position
should be an ARRAY_REF as the X,Y coordinates of the point to "tap".
* bounds
should be an ARRAY_REF of a bounding rectangle of the widget to tap.
Which contains two ARRAY_REFs for the top-left and bottom-right
coordinates, e.g. [ [tlX,tlY], [brX,brY] ] . This is convenient when
the widget is extracted from an XML dump of the UI (see
"dump_current_screen_ui($params)") which contains exactly this
bounding rectangle.
It returns 0 on success, 1 on failure.
input_text($params)
Dump the GPS / geo-location position for the device from its various
providers, if enabled.
electric-sheep-dump-current-location.pl --configfile config/myapp.conf --output geolocation.json
electric-sheep-emulator-geofix.pl
Set the GPS / geo-location position to the specified coordinates.
electric-sheep-dump-ui.pl --configfile config/myapp.conf --latitude 12.3 --longitude 45.6
electric-sheep-dump-screen-shot.pl
Take a screenshot of the device (current screen) and save to a PNG
file.
electric-sheep-dump-screen-shot.pl --configfile config/myapp.conf --output screenshot.png
electric-sheep-dump-screen-video.pl
Record a video of the device's current screen and save to an MP4 file.
electric-sheep-dump-screen-video.pl --configfile config/myapp.conf --output video.mp4 --time-limit 30
electric-sheep-pull-app-apk.pl
Extract the APK file (java bytecode) for an app installed on the device
and save locally, perhaps, for disassembly and/or modification and/or
re-installation.
electric-sheep-pull-app-apk.pl --package calendar2 --wildcard --output anoutdir --configfile config/myapp.conf --device Pixel_2_API_30_x86_
electric-sheep-install-app
Install an APK file onto the device, passing extra installation
parameters -r (for re-install) and -g (for granting permissions),
electric-sheep-install-app --apk-filename test.apk -p '-r' -p '-g' --configfile config/myapp.conf --device Pixel_2_API_30_x86_
electric-sheep-viber-send-message.pl
Send a message using the Viber app.
electric-sheep-viber-send-message.pl --message 'hello%sthere' --recipient 'george' --configfile config/myapp.conf --device Pixel_2_API_30_x86_
This one saves a lot of debugging information to debug which can be
used to deal with special cases or different versions of Viber:
electric-sheep-viber-send-message.pl --outbase debug --verbosity 1 --message 'hello%sthere' --recipient 'george' --configfile config/myapp.conf --device Pixel_2_API_30_x86_
TESTING
The normal tests under the t/ directory, initiated with make test
command, are quite limited in scope because they do not assume a
connected device. That is, they do not check any functions which
require interaction with a connected device.
The live tests under the xt/live directory, initiated with make
livetest command, require an Android emulator or real device (the
latter is not recommended) connected to your desktop computer on which
you are doing the testing. Note that testing with your smartphone is
not a good idea, please do not do this, unless it is some phone which
you do not store important data. It is very easy to get an emulated
Android device running on any OS.
So, prior to make livetest make sure you have an android emulator up
and running with, for example, emulator -avd Pixel_2_API_30_x86_ . See
section "Android Emulators" for how to install, list and run them
buggers.
At least one of the author tests under the xt/author directory,
initiated with make authortest command, require an APK file (to be
installed on the connected device) which is quite large and it is not
included in the distribution bundle of this module. Anyway, it is not a
good idea to install an unknown APK to your device. But if you want to
make this test then pull an APK of an existing app on your connected
device with electric-sheep-pull-app-apk.pl and point the test file to
this APK.
Testing will not send any messages via the device's apps. E.g. the
plugin Android::ElectricSheep::Automator::Plugins::Apps::Viber will not
send a message via Viber but it will mock it.
The live tests will sometimes fail because, so far, something
unexpected happened in the device. For example, in testing sending
input text to a text-edit widget, the calendar will be opened and a new
entry will be added and its text-edit widget will be targeted. Well,
sometimes the calendar app will give you some notification on startup
and this messes up with the focus. Other times, the OS will detect that
some app is taking too long to launch and pops up a notification about
"something is not responding, shall I close it". This steals the focus
and sometimes it causes the tests to fail.
PREREQUISITES
Android Studio
This is not a prerequisite but it is highly recommended to install it
(from https://developer.android.com/studio) on your desktop computer
because it contains all the executables you will need, saved in a well
documented file system hierarchy, which can then be accessed from the
command line. You will not be using the IDE or anything, just the
accompaniying binaries and libraries it comes with.
Additionally, Android Studio offers possibly the easiest way to create
Android Virtual Devices (AVD) which emulate an Android phone of various
specifications, phone models and sizes, API levels, etc. I mention this
because one can install apps on an AVD and control them from your
desktop as long as you are able to receive sms verification codes from
a real phone. Perhaps you will need an Android emulator image which
comes with Google Play Services, if you are installing apps from their
store. This is great for experimenting without plugging in your real
smartphone on your desktop.
The bottom line is that by installing Android Studio, you have all the
executables you need for running things from the command line and,
additionally, you have the easiest way for creating Android Virtual
Devices, which emulate Android devices: phones, tablets, automotive
displays. Once you have this set up, you will not need to open Android
Studio ever again unless you want to update your kit. All the
functionality will be accessible from the command line.
ADB
Android Debug Bridge (ADB) is the program which communicates with your
smartphone or an Android Virtual Device from your desktop (Linux, osx
and the unnamed 0$).
If you do not want to install Android Studio, the adb executable is
included in the package called "Android SDK Platform Tools" available
from the Android official site, here:
https://developer.android.com/tools/releases/platform-tools#downloads
You will need the adb executable to be on your path or specify its
fullpath in the configuration file supplied to
Android::ElectricSheep::Automator's constructor.
USB Debugging
The targeted smartphone must have "USB Debugging" enabled via the
"Developer mode". This is not to be confused with 'rooted' or
'jailbroken' modes, none of these are required for experimenting with
the current module.
In order to enable "USB Debugging", you need to set the smartphone to
enter "Developer" mode by following this procedure:
Go to Settings->System->About Phone Tap on Build Number 7 times [sic!].
Enter your phone pin and you are in developer mode.
You can exit Developer Mode by going to Settings->System->Developer and
turn it off. It is highly advised to turn off Developer Mode for
everyday use of your phone. Do not connect your smartphone to public
WIFI networks with Developer Mode ON.
Do not leave home with Developer Mode ON.
Once you have enabled "USB Debugging", you have two options for making
your device visible to your desktop and, consequently, to ADB and to
this module:
* connect your android device via a USB cable to your desktop
computer. I am not sure if you also need to tap on the USB charging
options and allow "Transfer Files".
* connect your device to the same WIFI network as your desktop
computer. Then follow instructions from, e.g., here
https://developer.android.com. This requires a newer Android version.
Android Emulators
It is possible to do most things your smartphone does with an Android
Virtual Device. You can install apps on the the virtual device which
you can register by supplying your real smartphone number.
List all virtual devices currently available in your desktop computer,
with emulator -list-avds which outputs something like:
Pixel_2_API_27_x86_
Pixel_2_API_30_x86_
Start a virtual device with emulator -avd Pixel_2_API_30_x86_
And hey, you have an android phone running on your desktop in its own
space, able to access the network but not the telephone network (no SIM
card).
It is possible to create a virtual device from the command line. But
perhaps it is easier if you download Android Studio from:
https://developer.android.com/studio and follow the setup there using
the GUI. You will need to do this just once for creating the device,
you can then uninstall Android Studio.
Android Studio will download all the required files and will create
some Android Virtual Devices (the "emulators") for you. It will also be
easy to update your stack in the future. Once you have done the above,
you no longer need to run Android Studio except perhaps for checking
for updates and all the required executables by this package will be
available from the command line.
Otherwise, download "Android SDK Platform Tools" available from the
Android official site, here:
https://developer.android.com/tools/releases/platform-tools#downloads
(this download is mentioned in ADB if you already fetched it).
Fetch the required packages with this command:
sdkmanager --sdk_root=/usr/local/android-sdk "platform-tools"
"platforms;android-30" "cmdline-tools;latest" "emulator"
Note that sdkmanager --list will list the latest android versions etc.
Now you should have access to avdmanager executable (it should be
located here:
/usr/local/android-sdk/cmdline-tools/latest/bin/avdmanager) which you
can use to create an emulator.
List all available android virtual devices availabe to you to create:
avdmanager list target
List all available devices you can emulate: avdmanager list device
List all available devices which have been created already and are
available to boot right now: avdmanager list avd
Create virtual device: avdmanager create avd -d "Nexus 6" -n myavd -k
"system-images;android-29;google_apis;x86"
(source: https://stackoverflow.com/a/77599934)
In Linux, the Android emulator image files are stored at
~/.config/.android/avd and/or at ~/.android/avd Each image consists of
an .avd file and a .ini file. As said before, you can boot a device
with emulator -avd 'Pixel_9' (the images will be Pixel_9.avd and
Pixel_9.ini)
NO ACCESS TO GOOGLE PLAY?
See here if your Android Emulator has no access to Google's App Store:
https://stackoverflow.com/questions/71815181/how-can-i-get-google-play-to-work-on-android-emulator-in-android-studio-bumblebe
Your mileage will lean on the low side.
( run in 0.605 second using v1.01-cache-2.11-cpan-483215c6ad5 )