Page History

Introduction

The tool lsst-build recursively clones, builds, and installs the LSST Stack. While the motivation for its creation was to serve as a backend for buildbot, anyone may download and run it to do stack-wide integration builds (including building with alternative versions of external packages, etc.). The core tool lives in lsst/lsst_build.git.

However, the easiest way to set up a personal build environment is to clone the lsst/lsstsw.git repository: 

...

...

git clone https://github.com/lsst/lsstsw.git

The lsstsw package is primarily used to manage the buildbot back-end directory on lsst-dev (aka ~lsstsw), but it's self-contained and can be used anywhere by anyone.

After the clone, run the following:

cd lsstsw
./bin/deploy

to bootstrap the environment. This will: 

• download and install Anaconda in ./anaconda
• download and install git in ./lfs
• download and install EUPS in ./eups
• make an empty stack directory in ./stack, with a default manifest.remap
• run git clone lsst_build master
• run git clone versiondb master

Once the bootstrap completes, set up the environment as directed by the script's output; e.g.: 

export LSSTSW=<where you've set it up>
export EUPS_PATH=$LSSTSW/stack
. $LSSTSW/bin/setup.sh

Note: you only need to run deploy once, although it's clever enough to turn into a no-op when it detects it has successfully run already.

Recursively cloning/rebuilding a package

Assuming you've done all of the above, simply run:

rebuild lsst_apps

lsst_apps is an empty package that depends on all of the LSST packages. You can use the "rebuild" command to build other packages by name as well. 

eups tags --clone b6132 current

• Search for and clone the package from our git repositories (as configured in etc/repos.yaml) into $LSSTSW/build
• Recursively clone all of its dependencies (also into $LSSTSW/build)
• Recursively build all its dependencies bottom-up, installing the built packages into $LSSTSW/stack, using an eups tag of the form bNNN (where N is a digit), e.g. b6132
• Build the package as well, and install it into $LSSTSW/stack

Note: rebuilding afwdata may take awhile, since it must download a few GB of data. If you already have copy that is checked out from the same URL as in the etc/repos.yaml file, you can use it by making a symlink to $LSSTSW/build/afwdata. In fact it is very useful to store afwdata outside of the lsstsw directory and symlink to it, as it allows you to delete your copy of lsstsw at any and start fresh, without downloading afwdata again. If you don't want a copy of afwdata at all then you can add it to the etc/exclusions.txt file.

Customizations are possible by editing the etc/settings.cfg.sh file, or by running lsst-build manually. See the documentation in bin/rebuild to see how.

To rebuild the entire stack, pick one of the top-level packages (e.g., lsst_distrib or lsst_apps).

Building branches

You can build specific branch(es) by running:

rebuild -r branch1 -r branch2 -r ... <packagename>

Before building, the code above will attempt to checkout branch1 (both in the package and its dependencies), and fall back to branch2 if it doesn't exist, and then fall back to master (or another default branch configured in etc/repos.yaml). This is what buildbot uses internally for forced builds of branches.

Other command-line options for rebuild

-p will clone the required packages and then stop, without building anything.

-u will bring over a current copy of etc/repos.yaml before starting the build. This can be handy if repositories have moved or been added and is used by our continuous integration system.

-t <tag> is deprecated; use this instead: eups tags --clone=oldtag newtag

Low(er)-level tool: lsst-build

Here is an example of how to run lsst-build (note: this omits some key setup details, for that see the package README):

lsst-build prepare \
  --exclusion-map=exclusions.txt \
  --version-git-repo=versiondb \
  ./build lsst_distrib
lsst-build build ./build

...

lsst-build prepare \
  ... \
  --ref tickets/1234 --ref next --ref master \
  build lsst_distrib

...

1. The new lsst-dev stack is in ~lsstsw/stack. Set your EUPS_PATH to point to it.
2. lsst-build right now periodically runs from cron and builds the master any time it changes. The results end up in ~lsstsw/stack. each build is EUPS-tagged with a unique build number (e.g., b1, b2, b3, ...). The latest build gets EUPS tagged as 'current'. There's no more need to run submitRelease, since everything is available.
3. when we want to release the stack, someone with ~lsstsw access will log into ~lsstsw and runs the standard 'eups distrib create', possibly EUPS-tagging it as something more memorable than bNNN (e.g., Winter2014). If it's useful, we could also automatically release the bNNN builds. 

...

lsst-build request-build tickets/1234 next master

The output could be e-mailed or shown on the console). 
Links:
  * https://github.com/lsst/lsst_build.git
    (look for the README in here)
  * https://github.com/lsst/versiondb.git