Launch an EC2 Instance. In Step 1 phase pick the Amazon Linux 2 image from the list of default AMIs. Two issues that should with Amazon Linux 2 are leaving the EPEL repositories permanently activated, as from experience this tends to interfere with normal functionality of the AMI, and interfering with PYTHONPATH, as from experience that interferes with the LSST Stack. This avoids those issues.

The end result should be an instance that has all the necessary dependencies to cleanly install the Stack, Pegasus and Condor just by following their official instructions.

1.1. Procedure


Get up to date

sudo yum update
sudo yum upgrade
sudo reboot now


Get correct toolkit:

sudo yum groupinstall "Development Tools"
sudo yum install cmake


Test if the default targets changed (at the time of writing the instructions outputs should be 7+):

g++ -v
gcc -v
gfortran -v


Add EPEL repository and install python-amqplib only, do not enable generally:

sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm


This is required package by Pegasus, I don't know its initialization procedure so instead of putting it in condor (like boto3, moto and psycopg2 for the stack) we install it system-wide::

sudo yum --enablerepo epel install python-amqplib


This us required package for HTCondor. Ensure this is outside of any type of virtual env, so that target python is the system python:

sudo yum --enablerepo epel install python-pip
sudo pip install --upgrade pip
sudo pip install awscli


Get some practical tools for work that you want. Vim exists, add Emacs and multiplexer of your choice (or stick with screen):

sudo yum install emacs tmux


Reboot to ensure that all updates have consolidated:

sudo reboot now

1.2. Installing the LSST Stack

Follow the official newinstall instructions, do not follow the system requirements step but continue straight to step 3., and at the end, while in the activated LSST environment, install

conda install boto3 moto psycopg2 awscli

until such a time these become a part of the LSST Stacks miniconda environment.


Potential awscli installation issues

Pegasus will expect `awscli` to exist in whichever environment it's launched. The base environments are prepared for system-wide installation. Setting up the LSST environment will overwrite the system-wide installation of the package. Unfortunately for w38 build of the stack one might experience the following error:

$ conda install -c conda-forge awscli
Solving environment: failed

CondaValueError: Malformed version string '~': invalid character(s).

related to a bug that was fixed in newer conda version. Since updating conda would update several different packages on which that particular weekly might depend on to avoid issues run

pip install awscli

instead.


Potential stack installation issues

If the LSST Stack was installed after Pegasus was already installed or `PYTHONPATH` was edited and the following, or similar, error occurs:

Traceback (most recent call last):
  File "/home/centos/w_2019_38/eups/2.1.5/bin/eups_setup", line 28, in <module>
    import eups.setupcmd
ModuleNotFoundError: No module named 'eups.setupcmd'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/centos/w_2019_38/eups/2.1.5/bin/eups_setup", line 42, in <module>
    import eups.setupcmd
ModuleNotFoundError: No module named 'eups.setupcmd'
-bash: export: setup: not a function
-bash: export: unsetup: not a function

run `unset PYTHONPATH` to proceed with the installation and export the additional `PYTHONPATH` values after. Note that the required setup for a system-wide installation of Pegasus, such as the Setting up HTCondor and condor_annex one, the following environmental variables are expected to exist for Pegasus to work within the LSST Stack environment:

export PEGASUS_HOME=/usr/
export PATH=${PEGASUS_HOME}/bin:$PATH
export PYTHONPATH=${PEGASUS_HOME}/lib64/python2.7/site-packages/:$PYTHONPATH

Because environment is order-sensitive to how these steps are executed in the full environment setups (condor, pegasus and stack) are done at login time in `~/.bashrc`. If logging in to an not-set-up environment is desired PYTHONPATH edit is required to be performed manually or by adding it to the loadLSST.* script manually (PEGASUS_HOME and PATH can be left in the `~./bashrc` and source & setup steps can be removed.



  • No labels