The general philosophy of the LSST software stack is that some python dependencies are so ubiquitous in scientific computing environments that it is more difficult for users to provide new versions than it is to supply them with the stack. The simulations framework relies on the following dependencies that are not part of the LSST stack:
- python 2.7
- Cython (note: even if using the stack-supplied Anaconda distribution, you should run conda install cython to install the Sims stack)
- git-lfs is required for installation from source (eups or lsstsw) – read-only mode is sufficient for installing the sims packages
- See http://developer.lsst.io/en/latest/tools/git_lfs.html for instructions in installing and setting up git-lfs for LSST
The recommended install instructions will provide these dependencies for you through an Anaconda distribution.
NOTE : Installation of all of lsst_sims (not just sims_maf) includes the installation of GalSim, which is currently unable to install on MacOS El Capitan. If you require the full sims package and GalSim and are using El Capitan, please contact Scott Daniel.
2) Add the LSST software channel
3) Install the desired sims packages
for just MAF
Then, every time you open a new shell, you can setup the sims packages using the following code. Note that the file `eups-setups.sh` is in `$ROOT/bin` where `$ROOT` is the conda environment that you installed into or if you did not specifically setup an environment (following the above method) it points to the conda installation. However, the location of the file does not matter, as the `ROOT/bin` directory should be in your path if you are using conda, and then you can launch the following commands from any working directory.
to later update use
The recommended (simplest) approach is to use the newest W14 LSST software stack, and use the LSST-provided version of anaconda python. (See here if you want to use your own python).
1. Start by installing the necessary parts of the LSST software stack and the LSST-installed anaconda.
In these instructions we assume you are installing in ~/lsst, however the install directory can be any place in the file system, including a place visible to all users. To install in another location, replace ~/lsst with the desired path in the following instructions. For multi user stacks, permissions are typically restricted to read only for the main stack so packages are not accidentally installed in the main stack.
Choose yes when prompted to install Anaconda (the recommended approach). Choose yes when prompted to install git**. These packages will not interfere with your system installed versions. This step will download about 1 GB of files to your machine and usually takes less than 15 minutes.
** Only choose 'yes' to install git if your current version is prior to git 1.7 or you have curl-devel installed on your system.
2. Set up the environment and install the simulations code and data.
This will install all packages currently in the catalogs simulations framework (CatSim) and metrics analysis framework and all dependencies. The installation should take on the order of 2 hours, with a final required installation size of 10GB.
Currently the complete list of installed packages is:
- base – LSST import utilities
- boost– Third party C++ package
- cfitsio– FITSIO I/O library
- daf_base – LSST data access framework
- doxygen– Documentation suite
- freetds– OpenSource implementation for Tabular Data Streams (needed by pymssql)
- healpy– Python bindings for HealPix
- palpy– Python binidings for PAL (Positional Astronomy Library)
- pex_config – LSST configuration package
- pex_exceptions – LSST exception handling package.
- pex_policy – Another LSST configuration package
- pyephem– Ephemeris generation code
- pyfits– Python bindings for interacting with FITS files
- pykg_config– Pure python implementation of pkg_config needed to install healpy on Mac OSX
- pymssql– Python bindings for talking to MS-SqlServer databases (needed to access UW base catalogs)
- scons– Software construction tool
- sconsUtils – LSST utilities for scons
- sims_catalogs_generation – Code for querying databases for catalog data
- sims_catalogs_measures – Code for constructing observed catalogs
- sims_catUtils – Package containing example code and definitions of base catalogs hosted at UW.
- sims_photUtils – Utilities for calculating photometry including variability
- sims_coordUtils – Utilities for calculating coordinate corrections (proper motion, parallax, refraction, etc.)
- sims_dustmaps – SFD dust maps
- sims_maf – Metrics Analysis Framework
- sims_sed_library – Library of SED data for simulated catalogs
- swig– Library for auto-generating python bindings for C++ code
- throughputs – Nominal throughput curves for SDSS and LSST systems
- utils – LSST utilities package
Any of the above packages and all their dependencies can be installed by replacing lsst_sims with the appropriate package name in the above code snippet (e.g. sims_maf). Installation is now complete. See package specific pages for documentation.
4. Setup installed packages
You have now downloaded and built all of the packages in lsst_sims. These packages are designed to be totally self-contained. They have been built in the directories
where yourOperatingSystem is 'DarwinX86' for Mac users and 'Linux64' for Linux users. In order to use the packages you installed, you must add these directories to your $PYTHONPATH. This is done using
setup is an eups command. your_package_name is the name of one of the packages you have installed (e.g. sims_maf or sims_catUtils). your_package_tag is a tag eups uses to keep track of the versions of each package you have built on your machine. This corresponds to the argument of '-t' in the 'eups distrib install' command above. So, if you wanted to setup the version of MAF you just downloaded, you would use
When you setup a package, eups inspects it and determines what other packages it depends on. Eups will also setup those prerequisite packages, preferring versions tagged with the tag you specified and defaulting to versions with the tag 'current'. If you do not want to default to 'current', you can specify more than one tag.
will setup sims_maf and all of its dependencies, preferring versions tagged as 'sims,' using versions tagged as '$USER' if no 'sims' version exists, and only using 'current' if neither 'sims' nor '$USER' can be found. This system of eups tags allows you to have multiple versions of the stack built on your system simultaneously. You will only ever be using the one that eups has setup. To see which versions of a package exist on your system (and which has been setup) use
- If you have issues with installation, first check that your system meets the minimum requirements listed here: LSST software user guide prerequisites.
- If you are having issues specifically with pyephem or healpy on a Mac, check for the existence of a /Developer directory. This is obsolete after upgrading to newer versions of XCode, but not removed by the XCode installer. Rename the /Developer directory and pyephem will install.
- If you are using your own python, be sure to check the Using Your Own Python page.
- If you are using your own anaconda python, be sure it is not installed in /anaconda (install somewhere like your home directory instead). This should be fixed in a new release soon.
- On a Mac, make sure you have accepted the terms on XCode. You can do this by opening the Xcode.app (should be in your Applications folder).
- git can fail, complaining about not having an https helper. If your native git version is > 1.7, you can probably use that rather than the LSST installed git.
- If all else fails, it's usually an issue with some environment variables interfering with the installation. You can create a new user and install the stack there. You can quickly login/out of a new user account as follows: First make a new admin-level user in System Preferences->Users and Groups, and then click on your name in the top right hand corner of the screen. A drop-down menu should appear, offering you a choice of other users to log in as. You might have to toggle the check box in System Preferences->Users and Groups->Login Options first though.
Mixing Installed Stack with Development Repositories
If you are going beyond simply using software packages provided by LSST and need a direct git clone of a particular repository (because you wish to contribute development work directly back into repository or because a new feature is available but has not been released officially yet), you can mix an installed stack with development repositories. For pure python packages, this is straightforward. The following steps will put a local copy of the sims_maf git repository into a pre-existing stack. **Note you do not have to do this just to install and use any sims package, such as MAF.
All packages may be declared with a version and a tag. The version distinguishes on instance of a particular package from another. The eups tag can be used to define a coherent set of packages. The philosophy is to tag all personal package (packages downloaded via git) with a custom tag using the username. Using this workflow allows packages to be set up with the -t $USER switch which means that custom packages are used if they are declared and main stack packages are used otherwise.
1. Move to a directory to hold the working repository and clone it:
Note that you will need a password on the stash server (or have set up ssh keys) to push to the server.
2. Declare and build the package:
You can check running the following that your dev version is being used by the following method.
See here for the confluence question dealing with how to be polite in a shared stack.
Note that when you are trying to simply use your git cloned version of the package, rather than install it, you simply need to 'setup sims_maf -t $USER'.
3. Code, commit and push
Code reviews should be handled by branching the repository and issuing a pull request through stash.