Page History

...

1. If you register, you will be given a piece of paper with your login information.  The information on the paper should contain:
   

   Credentials	EXAMPLE
   Login name	aws-lsstXX
   Password	Oh19-EXAMPLE
   AWS Access Key ID	AKIAIOSFODNN7-EXAMPLE
   AWS Secret Access Key	wJalrXUtnFEMI/K7MDENG/bxp-Fake-EXAMPLEKEY

   You need the Login name and Password in the next step, will use the AWS SSH Keys later. 
   
   

2. Navigate, in your internet browser of choice, to https://sanpadhi.signin.aws.amazon.com/console and sign in. If you have a password manager, it may try to auto-fill your personal Amazon credentials, but we want to use login credentials specific to today's tutorial.
   1. Leave the "Account ID or Alias" unchanged (it should read "sanpadhi" who is kindly facilitating this tutorial).
      
   2. Type your unique Login name in the "IAM user name" field.
   3. Type the shared Password in the "Password" box.
      
3. You will be redirected to the AWS Dashboard. We'll be using the EC2 (Elastic Compute) service to create a kind of virtual server, called an instance.
   1. Click on the "Services" tab at the top, next to the AWS logo.
   2. Type "EC2" in the search bar, and select EC2 by clicking on it.
      
      
4. You will be redirected to the Elastic Compute Dashboard.
   1. On the left hand side there is a panel with options. Under the "Image" section, select "AMIs."
      
      
      
5. You will be redirected to the Amazon Machine Image selection page. We already have a "private" AMI (Amazon Machine Image) ready to go for this demo. The default AMIs shown on this page are ones you created and own, so you won't see any at first.
   1. Click on the drop-down triangle and select "Private Images."
      
   2. Find the AMI named "demo_master" (ami-07d6c4c78c1530ff8) and click on it to select it. It has the friendly suggestion "Launch me!" as a name.
   3. Click the blue "Launch" button.
      
      
6. You will be redirected to the Launch Instance Wizard. The Wizard allows you to configure a variety of things about your instance. You will see many different instance types, and a "micro" free-tier option may be pre-selected.
   1. Scroll way down and select m5.xlarge for this demo. This instance has 4 CPUs and 16 GB RAM. You could select a different one if you prefer; the specs are different so the performance varies.
   2. On the top of the Launch Instance Wizard page, in blue numbered text, you can see various different machine configurations you can select. Feel free to browse through them; most of the defaults are OK.
   3. We need to adjust the security settings before launching. Click on "6. Configure Security Group," choose "Select an existing security group," and select the "lsst-demo" (sg-058badbfff072b4ae) security group.
   4. Click the big blue "Review and Launch"!
      
      
      
7. You will be redirected to Wizard Review page. It will warn you "Your instance is not eligible for the free usage tier!" and "Improve your instances' security. Your security group, lsst-demo, is open to the world." This is fine.
   
   1. Do a quick check of your setup before launching your instance. Ensure you have selected the "lsst-demo" security group! If not, you can edit it from here. 
   2. Click the big blue "Launch"  button in the bottom right of the screen!
      
      
8. A pop-up window will appear. This window allows you to configure the SSH keys you want to use to connect to the instance.
   1. Since you are using this account for the first time select "Create a new key pair".
      
   2. Enter a Key pair name. We recommend you use your name; for example "Firstname-demo."
       
   3. Click "Download Key Pair" to download a <your name>.pem file.

   4. Store this .pem file somewhere safe and easy to remember. If you are running a Linux machine we recommend the `~/.ssh` directory. Most SSH clients will not allow you to use SSH key files if their file permissions are too permissive, so update the permissions to be user-read-only.

      Code Block
      mv ~/Downloads/Firstname-demo.pem ~/.ssh/. chmod 400 ~/.ssh/Firstname-demo.pem

      
      

   5. You are now ready to click in the big blue "Launch Instances" button in the bottom right of the pop-up window.
      
      
9. Your instance is now launching! Click on the ID of your instance and wait for the "Instance State" to turn green.
   
   
   
   1. (OPTIONAL) If you want, you can give your instance a name, just so it's easier to find it later after you navigate away. To add a name, hover your mouse on top of the name box, which is blank by default. Click on the pencil to add a name.  Please be careful only do so to your instance, not somebody else's.
      

10. Congratulations, your instance is now up and running! Click "Connect" for instructions on how to connect to it.
    

...

export OUT="aws-lsstlsstXX-MyName"  # Change it!!

To run the LSST pipeline task, copy and paste (or type in) the following command to the Terminal: 

pipetask -d 'visit.visit=903334 and detector.detector=22' \
         -b 's3://lsst-demo-pegasus/input/butler.yaml' \
         -p lsst.ip.isr \
         -i calib,shared/ci_hsc \
         -o $OUT \
         run \
         -t isrTask.IsrTask:isr \
         -C isr:/home/centos/configs/isr.py

...

You will find that the configuration consists of a datastore and a registry. The datastore is the place where the files actually live. In this demo, it points to an S3 Bucket called "lsst-demo-repo." The registry is a database that describes the datasets and their metadata. In this demo, it points to a PostgreSQL RDS instance called demo-registry at port 5432 and in that RDS instance the database called "cihsc". If you were running the processing on your local machine, these would just be standard paths to files on your computer instead of "s3" and "postgresql."

If you want to see the source code for the IsrTask, it is on GitHub here.

...

You should find a folder named after your output collection ("aws-lsstlsstXX-MyName" in the example above). This was created when you ran the ISR job. Click on your folder and continue clicking down, you will see
"postISRCCD" > 903334 > postISRCCD_903334_22_HSC_NUMBER.fits" (Note this path includes the visit and detector numbers we specified earlier! The last number is distinct.)

...

aws s3 cp s3://lsst-demo-repo/aws-lsstlsstXX-MyName/postISRCCD/903334/postISRCCD_903334_22_HSC_17NUMBER.fits .

(Change the path as appropriate for your outputs.)

...

python

from lsst.daf.butler import Butler
butler = Butler("/home/centos/butler.yaml", collection="aws-lsstlsstXX-MyName")  # Update for your collection name
exp = butler.get("postISRCCD", {"visit":903334, "detector":22, "instrument":"HSC"})

...

Getting a "FileNotFoundError" on the butler.yaml file? Do "aws s3 cp s3://lsst-demo-pegasus/input/butler.yaml ~/butler.yaml" to download a new one yaml config file or change the path to the location you store the file. 

...