view release on metacpan or  search on metacpan

bin/gbrowse_aws_balancer.pl  view on Meta::CPAN

multiple ports were supported for performance reasons in earlier
single-threaded versions of the slave.

=item region (required for local masters)

The Amazon region in which to launch slaves. When the master is
running in EC2, this is automatically chosen to be the same as the
master's region and can be left blank.

=item image_id (required for local masters)

This is the ID of the AMI that will be used to launch slaves. The
correct value will be filled in when you run the
gbrowse_sync_aws_slave.pl. You can leave this value blank if the
GBrowse master is being run within an EC2 instance, in which case the
slave will be launched using the same AMI that was used to launch the
master.

=item data_snapshots (required for local masters)

Before launching the slave, attach EBS volumes created from one or
more volume snapshots listed in this option. Multiple snapshots can be
attached by providing a space-delimited list:

 data_snapshots = snap-12345 snap-abcdef

The gbrowse_sync_aws_slave.pl script will automatically maintain this
option for you.

=item availability_zone (optional)

This option will force the slave into the named availability zone. If
not specified, an availability zone in the current region will be
chosen at random.

=item subnet (optional)

If you are in a VPC environment, then this option will force the slave
into the named subnet. Ordinarily the balancer script will launch
slaves into non-VPC instances if the master is running on local
hardware or a non-VPC EC2 instance. The balancer will launch slaves
into the same VPC subnet as the master if the master is running on a
VPC instance.

=item security_group (optional)

This specifies the security group to assign the slaves to. If not
specified, a properly-configured security group will be created as
needed and destroyed when the balancer script exits. If you choose to
manage the security group manually, be sure to configure the firewall
ingress rule to allow access to the slave port(s) (see the "ports"
option) from the master's group or IP address.

=back

=head1 CONFIGURING AWS CREDENTIALS

To work, the balancer script must be able to make spot instance
requests and to monitor and terminate instances. To perform these
operations the script must have access to the appropriate AWS
credentials (access key and secret key) on the command line or as
environment variables. 

While the script does its best to shield the credentials from prying
eyes, there is still a chance that the credentials can be intercepted
by another party with login access to the machine that the master runs
on and use the credentials to run up your AWS bill. For this reason
some people will prefer to create an EC2 account or role with limited
access to AWS resources.

=over 4

=item 1. Your personal EC2 credentials

You may provide the balancer script with --access_key and --secret_key
command line arguments using your personal EC2 credentials or set the
environment variables EC2_ACCESS_KEY and EC2_SECRET_KEY. If not
provided, the script will interactively prompt for one or both of
these values.

This is the simplest method, but has the risk that if the credentials
are intercepted by a malicious third party, he or she gains access to
all your EC2 resources.

=item 2. The credentials of a restricted IAM account

You may use the Amazon AWS console to create an IAM (Identity Access
and Management) user with restricted permissions, and provide that
user's credentials to the script on the command line or with
environment variables. The following IAM permission policy is the
minimum needed for the balancer script to work properly:

 {
  "Statement": [
    {
      "Sid": "BalancerPolicy",
      "Action": [
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateSecurityGroup",
        "ec2:DeleteSecurityGroup",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeImageAttribute",
        "ec2:DescribeImages",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceAttribute",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVolumes",
        "ec2:DescribeSnapshots",
        "ec2:DescribeSpotInstanceRequests",
        "ec2:RequestSpotInstances",
        "ec2:CreateKeyPair",
        "ec2:DescribeKeyPairs",
        "ec2:DeleteKeyPair",
        "ec2:RunInstances",
        "ec2:TerminateInstances",
        "ec2:CreateSnapshot",
        "ec2:CreateVolume",
        "ec2:CreateTags",
        "ec2:DeleteTags"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
 }

Note that even with these restrictions, an unauthorized user with
access to the credentials could still launch a large number of spot
instances or terminate bona fide instances. This is just a fundamental
limitation of the granularity of EC2's permissions system.

=item 3. Create an IAM role

If the master is running on an EC2 instance, then the most convenient
way to pass credentials is by assigning the instance an IAM role. The
balancer script can then obtain temporary credentials by making
internal EC2 calls. The credentials do not need to be provided on the
command line or in environment variables, and are only valid for short
periods of time, limiting the effect of theft.

First, create an IAM role using the Amazon Console. Select
IAM->Roles->Create New Role, and give the role the name
"GBrowseMaster" (or whatever you prefer).

Next, when prompted for the role type, select AWS Service
Roles->Amazon EC2.

On the Select Role Permissions screen, choose "Custom Policy". Give
the policy a name like "GBrowseBalancer" and cut and paste into the
Policy Document text field the permission policy listed above in the
instructions for creating a restriced IAM account. Be sure to remove
the whitespace before the beginning of the first curly brace, or the
console will complain about an invalid policy.

You only need to do this once. After this, whenever you launch an
instance that will run the GBrowse master (typically from a GBrowse
AMI), specify the "GBrowseMaster" IAM role name. This can be done from
the AWS console's instance launch wizard, or by passing the -p option
to the ec2-run-instances command-line tool.

=back

=head1 USING THE INIT SCRIPT

The gbrowse-aws-balancer init script can be used on Ubuntu and
Debian-based systems to simplify launching the balancer at boot
time. It can be found in /etc/init.d by default, and is called in the
following manner:

start the service
 % sudo /etc/init.d/gbrowse-aws-balancer start

stop the service
 % sudo /etc/init.d/gbrowse-aws-balancer stop

stop and restart the service
 % sudo /etc/init.d/gbrowse-aws-balancer restart

show the status of the service (running, stopped)
 % sudo /etc/init.d/gbrowse-aws-balancer status

The various script options are all set in a single configuration file
named /etc/default/gbrowse-aws-balancer. The distribution contents of
this file looks like this:

 DAEMON=/usr/local/bin/gbrowse_aws_balancer.pl
 USER=www-data
 RUNDIR=/var/run/gbrowse
 LOGDIR=/var/log/gbrowse
 CONFFILE=/etc/gbrowse2/aws_balancer.conf
 ACCESS_KEY=YOUR_EC2_ACCESS_KEY_HERE
 SECRET_KEY=YOUR_EC2_SECRET_KEY_HERE
 VERBOSITY=3

The variables in this file set the location of the balancer script,
the location of its configuration file, the verbosity to run with, and
where to write the script's process ID and log information. In