Diff for "EC2Test/Image"

Not logged in - Log In / Register

Differences between revisions 17 and 31 (spanning 14 versions)
Revision 17 as of 2011-02-02 11:48:44
Size: 4442
Comment: clarify ec2 certificate generation
Revision 31 as of 2011-08-31 02:47:19
Size: 6472
Editor: wgrant
Comment: buildbot no longer uses ec2.
Deletions are marked like this. Additions are marked like this.
Line 21: Line 21:
The file `lib/devscripts/ec2test/account.py` contains a list `VALID_AMI_OWNERS`. You must either be one of those persons or add your account to the list. I am not sure if there should be some policy around who should be on that list. Only images from these account will be used by `bin/ec2`. By adding your account to a branch of yours, you can use your own private images, obviously, but nobody else can until this branch has been merged. The file `lib/devscripts/ec2test/account.py` contains a list `VALID_AMI_OWNERS`. You must either be one of those persons or add your account to the list. I am not sure if there should be some policy around who should be on that list. Only images from these account will be used by `bin/ec2`. By adding your account to a branch of yours, you can use your own private images, obviously, but nobody else can until this branch has been merged.  Add your [[https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key#account_identifiers|account ID]] as the key to the dictionary `VALID_AMI_OWNERS` and a human-readable identifier as the key.
Line 23: Line 23:
You also need to create a file named `~/.ec2/aws_user` and put your [[https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key#account_identifiers|account ID]] in there. In addition, you need to create a private key and X.509 certificate on the [[http://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key|AWS website]] - do this by clicking on the "X.509 Certificates" tab, then click "Create a new Certificate". It will give you the opportunity to download the private key and the certificate files, you need to store both files in `~/.ec2`. You also need to create a file named `~/.ec2/aws_user` and put your [[https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key#account_identifiers|account ID]] in there. In addition, you need to create a private key and X.509 certificate on the [[http://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key|AWS website]] - do this by clicking on the "X.509 Certificates" tab, then click "Create a new Certificate". It will give you the opportunity to download the private key and the certificate files, you need to store both files in `~/.ec2`.  You may only have one cert file in the .ec2 directory.
Line 37: Line 37:
The image is created based on a blank Lucid image. The name that the script will be looking for is ''launchpad-ec2testXXX'', where XXX is a revision counter that you need to increase when creating a new image. The current value can be found by running `bin/ec2 images`. The script will always use the image with the highest revision number it has access to. The image is created based on a blank Lucid image. The name that the script will be looking for is ''launchpad-ec2testXXX'', where XXX is a revision counter that you need to increase when creating a new image. The current value can be found by running `bin/ec2 images`. The script will always use the image with the highest revision number it has access to.  For example,
{{{
% bin/ec2 images
  Rev AMI Owner ID Owner
  520 ami-2b5e9e42 967591634984 jtv
  519 ami-15e8287c 292290876294 stevenk
  518 ami-92c53efb 424228475252 wallyworld
  517 ami-d8cc34b1 507541322704 sinzui
Line 39: Line 46:
Here is the command line which I ran in a current ''devel'' branch. Don't forget to increase the revision number. Also, the base image might change in the future. Please check if this page is up-to-date. }}}

The AMI ID (the bits after based-on:) in the command below is for the official Ubuntu 10.04 LTS (Lucid Lynx) Server 64-bit AMI found from http://uec-images.ubuntu.com/releases/10.04/release/. We've traditionally used the one marked `us-east-1, 64-bit, instance`. If you want to build an AMI using a different release, just use its ID there.

The original description for the image is [[http://aws.amazon.com/amis/4348|Ubuntu 10.04 LTS (Lucid Lynx) Server 64-bit AMI]], though that page does not show updates.

Based on the above output the command to run in a current ''devel'' branch follows. Note the revision number was increased. Also, the base image might change in the future. Please check if this page is up-to-date.
Line 42: Line 55:
bin/ec2 update-image --machine based-on:ami-da0cf8b3 --public launchpad-ec2test501 bin/ec2 update-image --machine based-on:ami-fbbf7892 --public launchpad-ec2test521
Line 45: Line 58:
The AMI ID (the bits after based-on:) above is for the official [[http://aws.amazon.com/amis/4348|Ubuntu 10.04 LTS (Lucid Lynx) Server 64-bit AMI]] (found from http://uec-images.ubuntu.com/releases). If you want to build an AMI using a different release, just use its ID there.
Line 52: Line 64:
 * '''The image will be private at first''' and you will have to set it to public manually to be used by everybody:
= Test the new image =

The image will be private at first and you will have to set it to public manually to be used by everybody. Before doing that you should test the image.
Run {{{bin/ec2 test}}} to ensure the image is good before making it public.

= Releasing the image for use =

Once the image is made public, and the branch adding your account to `VALID_AMI_OWNERS` has been merged, the image will become the default for the team to use.

Run `bin/ec2 images` and look for the ami-XXXXXX for your image in the second column of the output. Replace $ami in the command below with the ID:
Line 55: Line 77:
ec2-modify-image-attribute $ami --launch-permission -a all ec2-modify-image-attribute $ami --launch-permission -a all -C ~/.ec2/<your certificate file> -K ~/.ec2/<your private key file>
Line 64: Line 86:
== Update images for PQM and buldbot == == Update buildbot ==
Line 66: Line 88:
No use having your branches pass with the updated image but fail in the buildbot run. When the ec2test image is being updated, request an update of the images for PQM, lpbuildbot, and production by filing an RT with the LOSAs. Tom Haddon says they like to do them all at the same time to have them synchronized. [[https://bugs.edge.launchpad.net/launchpad-foundations/+bug/418549|Eventually this will not be necessary.]] No use having your branches pass with the updated image but fail in the buildbot run. When the ec2test image is being updated, you need to ensure that the buildbot slaves (pigeonpea and pilinut) have the changes made too; they're not cloud instances, so this needs to be done separately by a LOSA. This usually just consists of ensuring that the right version of launchpad-developer-dependencies is installed.
Line 72: Line 94:


== Deleting AMIs ==

Your AMI will one day become obsolete. If any of your images appear outside of the top 3 returned from `bin/ec2 images` then it's worth deleting it to save a few dollars a month. Here's how:

 1. Only the owner can delete his or her AMI. If you're not the owner, pester the owner.
 1. Find the bucket that corresponds to your AMI at https://console.aws.amazon.com/s3/home. The bucket will have a name of the form launchpad-ec2testXXX.
 1. Delete the contents of the bucket. Multiple objects in the bucket can be selected for deletion with a familiar click+shift-click.
 1. Delete the bucket.
 1. De-register the AMI at https://console.aws.amazon.com/ec2/home#s=Images.

That's it.

This page is documentation for updaters of the EC2 test image.

The EC2Test utility uses a special AMI to run the tests on. This image has all the dependencies pre-installed that are needed to run the Launchpad test suite. From time to time this image may need to be updated.

Why update the ec2test image?

A common reason for updating the image is that the launchpad-dependencies have changed and are needed for new branches to successfully pass the test suite. Once the updated dependencies have been uploaded to the LaunchpadPpa, the image needs to be updated to include the latest version.

Other reasons?

How to update the ec2test image

Here are the notes about what I had to do to accomplish this.

Be or become the right person

The file lib/devscripts/ec2test/account.py contains a list VALID_AMI_OWNERS. You must either be one of those persons or add your account to the list. I am not sure if there should be some policy around who should be on that list. Only images from these account will be used by bin/ec2. By adding your account to a branch of yours, you can use your own private images, obviously, but nobody else can until this branch has been merged. Add your account ID as the key to the dictionary VALID_AMI_OWNERS and a human-readable identifier as the key.

You also need to create a file named ~/.ec2/aws_user and put your account ID in there. In addition, you need to create a private key and X.509 certificate on the AWS website - do this by clicking on the "X.509 Certificates" tab, then click "Create a new Certificate". It will give you the opportunity to download the private key and the certificate files, you need to store both files in ~/.ec2. You may only have one cert file in the .ec2 directory.

EC2 API tools

Make sure you have the ec2-api-tools package installed. You will need the ec2-register command from that package (at least).

sudo apt-get install ec2-api-tools

Update the image

The image is created based on a blank Lucid image. The name that the script will be looking for is launchpad-ec2testXXX, where XXX is a revision counter that you need to increase when creating a new image. The current value can be found by running bin/ec2 images. The script will always use the image with the highest revision number it has access to. For example,

% bin/ec2 images
  Rev  AMI           Owner ID      Owner
  520  ami-2b5e9e42  967591634984  jtv
  519  ami-15e8287c  292290876294  stevenk
  518  ami-92c53efb  424228475252  wallyworld
  517  ami-d8cc34b1  507541322704  sinzui

The AMI ID (the bits after based-on:) in the command below is for the official Ubuntu 10.04 LTS (Lucid Lynx) Server 64-bit AMI found from http://uec-images.ubuntu.com/releases/10.04/release/. We've traditionally used the one marked us-east-1, 64-bit, instance. If you want to build an AMI using a different release, just use its ID there.

The original description for the image is Ubuntu 10.04 LTS (Lucid Lynx) Server 64-bit AMI, though that page does not show updates.

Based on the above output the command to run in a current devel branch follows. Note the revision number was increased. Also, the base image might change in the future. Please check if this page is up-to-date.

bin/ec2 update-image --machine based-on:ami-fbbf7892 --public launchpad-ec2test521

The script will run for some time, somewhere between 30-60 minutes. It installs all the dependencies in the base image, creates a new image of the result and uploads and registers it with AWS.

A couple of notes:

  • For some reason, you get asked whether you want to continue without installing GRUB when setting up the image. Just say "Yes" and everything should proceed normally.

Test the new image

The image will be private at first and you will have to set it to public manually to be used by everybody. Before doing that you should test the image. Run bin/ec2 test to ensure the image is good before making it public.

Releasing the image for use

Once the image is made public, and the branch adding your account to VALID_AMI_OWNERS has been merged, the image will become the default for the team to use.

Run bin/ec2 images and look for the ami-XXXXXX for your image in the second column of the output. Replace $ami in the command below with the ID:

ec2-modify-image-attribute $ami --launch-permission -a all -C ~/.ec2/<your certificate file> -K ~/.ec2/<your private key file>

Remember that your account must be in VALID_AMI_OWNERS for the script to actually use it.

What else?

Update buildbot

No use having your branches pass with the updated image but fail in the buildbot run. When the ec2test image is being updated, you need to ensure that the buildbot slaves (pigeonpea and pilinut) have the changes made too; they're not cloud instances, so this needs to be done separately by a LOSA. This usually just consists of ensuring that the right version of launchpad-developer-dependencies is installed.

Misc

Some people have experienced problems when the AMI was being setup (bzr timeouts); if that happens, just retry.

Deleting AMIs

Your AMI will one day become obsolete. If any of your images appear outside of the top 3 returned from bin/ec2 images then it's worth deleting it to save a few dollars a month. Here's how:

  1. Only the owner can delete his or her AMI. If you're not the owner, pester the owner.
  2. Find the bucket that corresponds to your AMI at https://console.aws.amazon.com/s3/home. The bucket will have a name of the form launchpad-ec2testXXX.

  3. Delete the contents of the bucket. Multiple objects in the bucket can be selected for deletion with a familiar click+shift-click.
  4. Delete the bucket.
  5. De-register the AMI at https://console.aws.amazon.com/ec2/home#s=Images.

That's it.

EC2Test/Image (last edited 2012-01-06 20:56:41 by gary)