Android-ElectricSheep-Automator
view release on metacpan or search on metacpan
# NAME
Android::ElectricSheep::Automator - Do Androids Dream of Electric Sheep? Smartphone control from your desktop.
# VERSION
Version 0.09
# WARNING
Current distribution is extremely alpha. API may change.
# SYNOPSIS
The present package fascilitates the control
of a USB-debugging-enabled
Android device, e.g. a real smartphone,
or an emulated (virtual) Android device,
from your desktop computer using Perl.
It's basically a thickishly-thin wrapper
to the omnipotent Android Debug Bridge (adb)
program.
**Note that absolutely nothing is
installed on the connected device,
neither any of its settings will be modified by this package**.
See ["WILL ANYTHING BE INSTALLED ON THE DEVICE?"](#will-anything-be-installed-on-the-device).
use Android::ElectricSheep::Automator;
my $mother = Android::ElectricSheep::Automator->new({
# optional as there is a default, but you may have
# problems with the location of the adb executable
'configfile' => $configfile,
'verbosity' => 1,
# we already have a device connected and ready to control
'device-is-connected' => 1,
});
# find the devices connected to desktop and set one.
my @devices = $mother->adb->devices;
$mother->connect_device({'serial' => $devices->[0]->serial})
or die;
# no device identification is required for the method call
# if there is only one connected device:
$mother->connect_device() if scalar(@devices)==0;
# Go Home
$mother->home_screen() or die;
# swipe up/down/left/right
$mother->swipe({'direction'=>up}) or die;
# dt is the time to swipe in millis,
# the shorter the faster the swipe
$mother->swipe({'direction'=>left, 'dt'=>100}) or die;
# tap
$mother->tap({'position'=>[100,200]});
# uses swipe() to move in screens (horizontally):
$mother->next_screen() or die;
$mother->previous_screen() or die;
# bottom navigation:
# the "triangle" back button
$mother->navigation_menu_back_button() or die;
# the "circle" home button
$mother->navigation_menu_home_button() or die;
# the "square" overview button
$mother->navigation_menu_overview_button() or die;
# open/close apps
$mother->open_app({'package'=>qr/calendar$/i}) or die;
$mother->close_app({'package'=>qr/calendar$/i}) or die;
# push pull files
$mother->adb->pull($deviceFile, $localFile);
$mother->adb->push($localFile, $deviceFileOrDir);
# guess what!
my $xmlstr = $mother->dump_current_screen_ui();
# Pull the apk(s) for an app from device and save locally
my $res = $mother->pull_app_apk_from_device({
package => 'com.google.android.calendar'
# or qr/calendar/i
'output-dir' => '/tmp/apks-of-calendar-app',
});
print $res->{'com.google.android.calendar'}->[0]->['local-path'};
# Install apk(s) for an app onto the device
$mother->install_app({
'apk-filename' => ['/tmp/apks/base.apk', '/tmp/apks/config.apk'],
# or just a string scalar '/tmp/apks/1.apk'
# optional params to the adb install command
'install-parameters' => ['-r', '-g']
});
# CONSTRUCTOR
## **`new($params)`**
Creates a new `Android::ElectricSheep::Automator` object. `$params`
is a hash reference used to pass initialization options which may
or should include the following:
- **`confighash`** or **`configfile`**
the configuration
file holds
configuration parameters. Its format is "enhanced" JSON
(see [Config::JSON::Enhanced](https://metacpan.org/pod/Config%3A%3AJSON%3A%3AEnhanced)) which is basically JSON
which allows comments between ` </* ` and ` */> `.
Here is an example configuration file to get you started:
{
"adb" : {
"path-to-executable" : "/usr/local/android-sdk/platform-tools/adb"
},
"debug" : {
"verbosity" : 0,
</* cleanup temp files on exit */>
"cleanup" : 1
},
"logger" : {
</* log to file if you uncomment this, else console */>
"filename" : "my.log"
}
}
All sections in the configuration are mandatory.
Setting `"adb"` to the wrong path will yield problems.
`confighash` is a hash of configuration options with
structure as above and can be supplied to the constructor
instead of the configuration file.
If no configuration is specified, then a default
configuration will be used. In this case please
specify **`adb-path-to-executable`** to point
to the location of `adb`. Most likely
the default path will not work for you.
- **`adb-path-to-executable`**
optionally specify the path to the `adb` executable in
your desktop system. This will override the setting
` 'adb'->'path-to-executable' ` in the configuration,
if it was provided. Use this option if you are not
providing any configuration and so the default configuration
will be used. But it will most likely fail because of this
path not being correct for your system. So, if you are going
to omit providing a configuration and the default configuration
will be used do specify the `adb` path via this option (but you
don't have to and your mileage may vary).
- **`device-serial`** or **`device-object`**
optionally specify the serial
of a device to connect to on instantiation,
or a [Android::ElectricSheep::Automator::DeviceProperties](https://metacpan.org/pod/Android%3A%3AElectricSheep%3A%3AAutomator%3A%3ADeviceProperties)
object you already have handy. Alternatively,
use ["connect\_device($params)"](#connect_device-params) to set the connected device at a later
time. Note that there is no need to specify a
device if there is exactly one connected device.
- **`adb`**
optionally specify an already created [Android::ADB](https://metacpan.org/pod/Android%3A%3AADB) object.
Otherwise, a fresh object will be created based
on the configuration under the `adb` section of the configuration.
- **`device-is-connected`**
optionally set it to 1
in order to communicate with the device
and get some information about it like
screen size, resolution, orientation, etc.
And also allow use of
functionality which needs communicating with a device
like ["swipe($params)"](#swipe-params), ["home\_screen($params)"](#home_screen-params),
["open\_app($params)"](#open_app-params), etc.
After instantiation, you can use the
method ["connect\_device($params)"](#connect_device-params) and
["disconnect\_device()"](#disconnect_device) for conveying
this information to the module.
Also note that if there are
more than one devices connected to the desktop, make sure
you specify which one with the `device` parameter.
Default value is 0.
- **`logger`**
optionally specify a logger object
to be used (instead of creating a fresh one). This object
must implement these methods: `info()`, `warn()`, `error()`.
[Mojo::Log](https://metacpan.org/pod/Mojo%3A%3ALog) fits perfectly.
- **`logfile`**
optionally specify a file to
save logging output to. This overrides the `filename`
key under section `logger` of the configuration.
- **`verbosity`**
optionally specify a verbosity level
which will override what the configuration contains. Default
is `0`.
- **`cleanup`**
optionally specify a flag to clean up
any temp files after exit which will override what the
configuration contains. Default is `1`, meaning Yes!.
# METHODS
Note:
- **`ARRAY_REF`** : `my $ar = [1,2,3]; my $ar = \@ahash; my @anarray = @$ar;`
- **`HASH_REF`** : `my $hr = {1=`1, 2=>2}; my $hr = \\%ahash; my %ahash = %$hr;>
- In this module parameters to functions are passed as a HASH\_REF.
Functions return back objects, ARRAY\_REF or HASH\_REF.
## **`devices()`**
Lists all Android devices connected to your
desktop and returns these as an ARRAY\_REF which can be empty.
It returns `undef` on failure.
## **`connect_device($params)`**
Specifies the current Android device to control. Its use is
required only if you have more than one devices connected.
`$params` is a HASH\_REF which should contain exactly
one of the following:
- **`serial`** should contain
the serial (string) of the connected device as returned
by ["devices()"](#devices).
- **`device-object`** should be
an already existing [Android::ElectricSheep::Automator::DeviceProperties](https://metacpan.org/pod/Android%3A%3AElectricSheep%3A%3AAutomator%3A%3ADeviceProperties)
object.
It returns `0` on success, `1` on failure.
## **`dump_current_screen_ui($params)`**
It dumps the current screen as XML and returns that as
a string, optionally saving it to the specified file.
`$params` is a HASH\_REF which may or should contain:
- **`filename`**
optionally save the returned XML string to the specified file.
It returns `undef` on failure or the UI XML dump, as a string, on success.
## **`dump_current_screen_shot($params)`**
It dumps the current screen as a PNG image and returns that as
a [Image::PNG](https://metacpan.org/pod/Image%3A%3APNG) object, optionally saving it to the specified file.
`$params` is a HASH\_REF which may or should contain:
- **`filename`**
optionally save the returned XML string to the specified file.
It returns `undef` on failure or a [Image::PNG](https://metacpan.org/pod/Image%3A%3APNG) image, on success.
## **`dump_current_screen_video($params)`**
It dumps the current screen as MP4 video and saves that
in specified file.
`$params` is a HASH\_REF which may or should contain:
- **`filename`**
save the recorded video to the specified file in MP4 format. This
is required.
- **`time-limit`**
optionally specify the duration of the recorded video, in seconds. Default is 10 seconds.
## **`install_app($params)`**
It installs the app from its specified
APK (bytecode archive and more) file.
`$params` is a HASH\_REF which should contain:
- **`apk-filename`**
The APK filename to install onto the device. It must
exist locally, obviously.
- **`install-parameters`**
Optional parameters to be passed on to the `adb install`
command. Nothing is expected here. Refer to the
[adb documentation](https://developer.android.com/tools/adb)
for what parameters are supported. For example, `-r` is for
re-installation of an existing app and retaining its
previous data.
It returns `1` on failure.
It returns `0` on success.
Note that there is a script available
which utilises this method, see [electric-sheep-install-app.pl](https://metacpan.org/pod/electric-sheep-install-app.pl).
## **`is_app_running($params)`**
It checks if the specified app is running on the device.
The name of the app must be exact.
Note that you can search for running apps / commands
with extended regular expressions using L/pgrep()>
`$params` is a HASH\_REF which should contain:
- **`appname`**
the name of the app to check if it is running.
It must be its exact name. Basically it checks the
output of [pidof()](https://metacpan.org/pod/pidof%28%29).
It returns `undef` on failure,
`1` if the app is running or `0` if the app is not running.
## **`find_current_device_properties($params)`**
It enquires the device currently connected,
and specified with ["connect\_device($params)"](#connect_device-params), if needed,
and returns back an [Android::ElectricSheep::Automator::DeviceProperties](https://metacpan.org/pod/Android%3A%3AElectricSheep%3A%3AAutomator%3A%3ADeviceProperties)
object containing this information, for example screen size,
resolution, serial number, etc.
It returns [Android::ElectricSheep::Automator::DeviceProperties](https://metacpan.org/pod/Android%3A%3AElectricSheep%3A%3AAutomator%3A%3ADeviceProperties)
object on success or `undef` on failure.
## **`connect_device()`**
It signals to our object that there is now
a device connected to the desktop and its
enquiry and subsequent control can commence.
If this is not called and neither `device-is-connected => 1`
is specified as a parameter to the constructor, then
the functionality will be limited and access
to functions like ["swipe($params)"](#swipe-params), ["open\_app($params)"](#open_app-params), etc.
will be blocked until the caller signals that
a device is now connected to the desktop.
Using ["connect\_device($params)"](#connect_device-params) to specify which device
to target in the case of multiple devices
connected to the desktop will also call this
method.
This method will try to enquire the connected device
about some of its properties, like screen size,
resolution, orientation, serial number etc.
This information will subsequently be available
via `$self->`device\_properties()>.
It returns `0` on success, `1` on failure.
## **`disconnect_device()`**
Signals to our object that it should consider
that there is currently no device connected to
the desktop (irrespective of that is true or not)
which will block access to ["swipe($params)"](#swipe-params), ["open\_app($params)"](#open_app-params), etc.
## **`device_properties()`**
It returns the currently connected device properties
as a [Android::ElectricSheep::Automator::DeviceProperties](https://metacpan.org/pod/Android%3A%3AElectricSheep%3A%3AAutomator%3A%3ADeviceProperties)
object or `undef` if there is no connected device.
The returned object is constructed during a call
to ["find\_current\_device\_properties($params)"](#find_current_device_properties-params)
which is called via ["connect\_device($params)"](#connect_device-params) and will persist
for the duration of the connection.
However, after a call to ["disconnect\_device()"](#disconnect_device)
this object will be discarded and `undef` will be
returned.
## **`swipe($params)`**
Emulates a "swipe" in four directions.
Sets the current Android device to control. It is only
required if you have more than one device connected.
`$params` is a HASH\_REF which may or should contain:
- **`direction`**
should be one of
- up
- down
- left
- right
- **`dt`**
denotes the time taken for the swipe
in milliseconds. The smaller its value the faster
the swipe. A value of `100` is fast enough to swipe to
the next screen.
It returns `0` on success, `1` on failure.
## **`tap($params)`**
Emulates a "tap" at the specified location.
`$params` is a HASH\_REF which must contain one
of the following items:
- **`position`**
should be an ARRAY\_REF
as the `X,Y` coordinates of the point to "tap".
- **`bounds`**
should be an ARRAY\_REF of a bounding rectangle
of the widget to tap. Which contains two ARRAY\_REFs
for the top-left and bottom-right coordinates, e.g.
` [ [tlX,tlY], [brX,brY] ] `. This is convenient
when the widget is extracted from an XML dump of
the UI (see ["dump\_current\_screen\_ui($params)"](#dump_current_screen_ui-params)) which
contains exactly this bounding rectangle.
## **`electric-sheep-dump-current-location.pl`**
Dump the GPS / geo-location position for the device from its various providers, if enabled.
electric-sheep-dump-current-location.pl --configfile config/myapp.conf --output geolocation.json
## **`electric-sheep-emulator-geofix.pl`**
Set the GPS / geo-location position to the specified coordinates.
electric-sheep-dump-ui.pl --configfile config/myapp.conf --latitude 12.3 --longitude 45.6
## **`electric-sheep-dump-screen-shot.pl`**
Take a screenshot of the device (current screen) and save to a PNG file.
electric-sheep-dump-screen-shot.pl --configfile config/myapp.conf --output screenshot.png
## **`electric-sheep-dump-screen-video.pl`**
Record a video of the device's current screen and save to an MP4 file.
electric-sheep-dump-screen-video.pl --configfile config/myapp.conf --output video.mp4 --time-limit 30
## **`electric-sheep-pull-app-apk.pl`**
Extract the APK file (java bytecode) for an app installed on the device and save locally, perhaps, for disassembly and/or modification and/or re-installation.
electric-sheep-pull-app-apk.pl --package calendar2 --wildcard --output anoutdir --configfile config/myapp.conf --device Pixel_2_API_30_x86_
## **`electric-sheep-install-app`**
Install an APK file onto the device, passing extra installation
parameters `-r` (for re-install) and `-g` (for granting permissions),
electric-sheep-install-app --apk-filename test.apk -p '-r' -p '-g' --configfile config/myapp.conf --device Pixel_2_API_30_x86_
## **`electric-sheep-viber-send-message.pl`**
Send a message using the Viber app.
electric-sheep-viber-send-message.pl --message 'hello%sthere' --recipient 'george' --configfile config/myapp.conf --device Pixel_2_API_30_x86_
This one saves a lot of debugging information to `debug` which can be used to
deal with special cases or different versions of Viber:
electric-sheep-viber-send-message.pl --outbase debug --verbosity 1 --message 'hello%sthere' --recipient 'george' --configfile config/myapp.conf --device Pixel_2_API_30_x86_
# TESTING
The normal tests under the `t/` directory, initiated with `make test` command,
are quite limited in scope because they do not assume
a connected device. That is, they do not check any
functions which require interaction with a connected
device.
The _live tests_ under the `xt/live` directory, initiated with
`make livetest` command, require
an Android emulator or real device (the latter **is not recommended**)
connected to your desktop computer on which you are doing the testing.
Note that testing
with your smartphone is not a good idea, please do not do this,
unless it is some phone which you do not store important data.
It is very easy to get an emulated Android device running on any OS.
So, prior to `make livetest` make sure you have an android
emulator up and running with, for example,
`emulator -avd Pixel_2_API_30_x86_` . See section
["Android Emulators"](#android-emulators) for how to install, list and run them
buggers.
At least one of the _author tests_ under the `xt/author` directory,
initiated with
`make authortest` command, require an APK file (to be installed on the connected
device) which is quite large and it is not included in the distribution
bundle of this module. Anyway, it is not a good idea to install
an unknown APK to your device. But if you want to make this test
then pull an APK of an existing app on your connected device
with [electric-sheep-pull-app-apk.pl](https://metacpan.org/pod/electric-sheep-pull-app-apk.pl) and point the test file
to this APK.
Testing will not send any messages via the device's apps.
E.g. the plugin [Android::ElectricSheep::Automator::Plugins::Apps::Viber](https://metacpan.org/pod/Android%3A%3AElectricSheep%3A%3AAutomator%3A%3APlugins%3A%3AApps%3A%3AViber)
will not send a message via Viber but it will mock it.
The live tests will sometimes fail because, so far,
something unexpected happened in the device. For example,
in testing sending input text to a text-edit widget,
the calendar will be opened and a new entry will be added
and its text-edit widget will be targeted. Well, sometimes
the calendar app will give you some notification
on startup and this messes up with the focus.
Other times, the OS will detect that some app is taking too
long to launch and pops up a notification about
"_something is not responding, shall I close it_".
This steals the focus and sometimes it causes
the tests to fail.
# PREREQUISITES
## Android Studio
This is not a prerequisite but it is
highly recommended to install it
(from [https://developer.android.com/studio](https://developer.android.com/studio))
on your desktop computer because it contains
all the executables you will need,
saved in a well documented file system hierarchy,
which can then be accessed from the command line.
You will not be using the IDE or anything, just
the accompaniying binaries and libraries it comes with.
Additionally, Android Studio offers possibly the
easiest way to create Android Virtual Devices (AVD) which emulate
an Android phone of various specifications, phone models and sizes,
API levels, etc.
I mention this because one can install apps
on an AVD and control them from your desktop
as long as you are able to receive sms verification
codes from a real phone. Perhaps you will need an Android
emulator image which comes with Google Play Services,
if you are installing apps from their store.
This is great for
experimenting without plugging in your real
smartphone on your desktop.
The bottom line is that by installing Android Studio,
you have all the executables you need for running things
from the command line and, additionally, you have
the easiest way for creating Android
Virtual Devices, which emulate Android devices: phones,
tablets, automotive displays. Once you have this set up, you
will not need to open Android Studio ever again unless you
want to update your kit. All the functionality
will be accessible from the command line.
## ADB
Android Debug Bridge (ADB) is the program
which communicates with your smartphone or
an Android Virtual Device from
your desktop (Linux, osx and the unnamed `0$`).
If you do not want to install Android Studio, the `adb` executable
is included in the package called
"Android SDK Platform Tools" available from
the Android official site, here:
[https://developer.android.com/tools/releases/platform-tools#downloads](https://developer.android.com/tools/releases/platform-tools#downloads)
You will need the `adb` executable to be on your path
or specify its fullpath in the configuration file
supplied to [Android::ElectricSheep::Automator](https://metacpan.org/pod/Android%3A%3AElectricSheep%3A%3AAutomator)'s constructor.
## USB Debugging
The targeted smartphone must have "USB Debugging" enabled
via the "Developer mode".
This is not
to be confused with 'rooted' or 'jailbroken' modes, none of
these are required for experimenting with the current module.
In order to enable "USB Debugging", you need
to set the smartphone to enter "Developer" mode by
following this procedure:
Go to `Settings->System->About Phone`
Tap on `Build Number` 7 times \[sic!\].
Enter your phone pin and you are in developer mode.
You can exit Developer Mode by going to
`Settings->System->Developer` and turn it off.
It is highly advised to turn off Developer Mode
for everyday use of your phone.
**Do not connect your smartphone
to public WIFI networks with Developer Mode ON**.
**Do not leave home with Developer Mode ON**.
Once you have enabled "USB Debugging", you have
two options for making your device visible to
your desktop and, consequently, to ADB and to this module:
- connect your android device via a USB cable
to your desktop computer. I am not sure if you also
need to tap on the USB charging options and allow
"Transfer Files".
- connect your device to the same WIFI network
as your desktop computer. Then follow instructions
from, e.g., here [https://developer.android.com](https://developer.android.com).
This requires a newer Android version.
## Android Emulators
It is possible to do most things your
smartphone does with an Android Virtual Device.
You can install apps on the the virtual device which
you can register by supplying your real smartphone
number.
List all virtual devices currently available
in your desktop computer, with `emulator -list-avds`
which outputs something like:
Pixel_2_API_27_x86_
Pixel_2_API_30_x86_
Start a virtual device with `emulator -avd Pixel_2_API_30_x86_`
And hey, you have an android phone running on your
desktop in its own space, able to access the network
but not the telephone network (no SIM card).
It is possible to create a virtual device
from the command line.
But perhaps it is easier if you download Android Studio
from: [https://developer.android.com/studio](https://developer.android.com/studio) and follow
the setup there using the GUI. You will need to do this just
once for creating the device, you can then uninstall Android Studio.
Android Studio will download all the
required files and will create some Android Virtual
Devices (the "emulators") for you. It will also be easy to
update your stack in the future. Once you have done the above,
you no longer need to run Android Studio except perhaps for
checking for updates and **all the required executables by this
package will be available from the command line**.
Otherwise, download "Android SDK Platform Tools" available from
the Android official site, here:
[https://developer.android.com/tools/releases/platform-tools#downloads](https://developer.android.com/tools/releases/platform-tools#downloads)
(this download is mentioned in [ADB](https://metacpan.org/pod/ADB) if you already fetched it).
Fetch the required packages with this command:
`sdkmanager --sdk_root=/usr/local/android-sdk "platform-tools" "platforms;android-30" "cmdline-tools;latest" "emulator"`
Note that `sdkmanager --list` will list the latest android versions etc.
Now you should have access to `avdmanager` executable
(it should be located here: `/usr/local/android-sdk/cmdline-tools/latest/bin/avdmanager`)
which you can use to create an emulator.
List all available android virtual devices availabe to you **to create**: `avdmanager list target`
List all available devices you can emulate: `avdmanager list device`
List all available devices which have been created already and are available to boot right now: `avdmanager list avd`
Create virtual device: `avdmanager create avd -d "Nexus 6" -n myavd -k "system-images;android-29;google_apis;x86"`
(source: [https://stackoverflow.com/a/77599934](https://stackoverflow.com/a/77599934))
In Linux, the Android emulator image files are stored
at `~/.config/.android/avd` and/or at `~/.android/avd`
Each image consists of an `.avd` file and a `.ini` file.
As said before, you can boot a device with `emulator -avd 'Pixel_9'`
(the images will be `Pixel_9.avd` and `Pixel_9.ini`)
# NO ACCESS TO GOOGLE PLAY?
See here if your Android Emulator has no access to Google's App Store:
[https://stackoverflow.com/questions/71815181/how-can-i-get-google-play-to-work-on-android-emulator-in-android-studio-bumblebe](https://stackoverflow.com/questions/71815181/how-can-i-get-google-play-to-work-on-android-emulator-in-android-studio-bumbl...
Your mileage will lean on the low side.
# USING YOUR REAL SMARTPHONE
Using your real smartphone
with such a powerful tool may not be such
( run in 0.497 second using v1.01-cache-2.11-cpan-d7a12ab2c7f )