This page will document how to make a third-party software package install-able using the eups distrib install command.

Creating the package

Repositories containing third-party packages exist in the LSST GitHub organization, github.com/lsst.  (Unfortunately, it is currently difficult to distinguish between an LSST package and a third-party package.  The list here and DM Third Party Software may help.)  In order to distribute a new third-party package, someone with administrator privileges will have to create a new repository of this form for you. Create a development branch on that repository and set it up to distribute the package as described below.  You will be able to test the package distribution off of your development branch before you merge to master.

The repository, once created needs to contain the following directories:

...

...

We discuss the contents of ups/ and patches/ in more detail below.

The ups/ directory

setupRequired(python)
setupRequired(pykg_config)
setupRequired(cfitsio)
setupRequired(pyfits)
envPrepend(PYTHONPATH, ${PRODUCT_DIR}/lib/python)
envPrepend(LD_LIBRARY_PATH, ${PRODUCT_DIR}/lib)
envPrepend(DYLD_LIBRARY_PATH, ${PRODUCT_DIR}/lib) 

export SCONSFLAGS=$SCONSFLAGS" USE_UNKNOWN_VARS=true TMV_DIR="$TMV_DIR" PREFIX="$PREFIX" PYPREFIX="$PREFIX"/lib/python EXTRA_LIB_PATH="$TMV_DIR"/lib EXTRA_INCLUDE_PATH="$TMV_DIR"/include"

# EupsPkg config file. Sourced by 'eupspkg'

prep()
{
	# Select the apropriate Anaconda distribution
	OS=$(uname -s -m)
	case "$OS" in
		"Linux x86_64")		FN=Anaconda-2.1.0-Linux-x86_64.sh ;;
		"Linux "*) 		FN=Anaconda-2.1.0-Linux-x86.sh ;;
		"Darwin x86_64")	FN=Anaconda-2.1.0-MacOSX-x86_64.sh ;;
		*) 			die "unsupported OS or architecture ($OS). try installing Anaconda manually."
	esac

	# Prefer system curl; user-installed ones sometimes behave oddly
	if [[ -x /usr/bin/curl ]]; then
		CURL=${CURL:-/usr/bin/curl}
	else
		CURL=${CURL:-curl}
	fi

	"$CURL" -s -L -o installer.sh http://repo.continuum.io/archive/$FN
}

build() { :; }

install()
{
	clean_old_install

	bash installer.sh -b -p "$PREFIX"

	if [[ $(uname -s) = Darwin* ]]; then
		#run install_name_tool on all of the libpythonX.X.dylib dynamic
		#libraries in anaconda
		for entry in $PREFIX/lib/libpython*.dylib
		do
			install_name_tool -id $entry $entry
		done
	fi

	install_ups
}

The patches/ directory

Sometimes, it will be necessary to change the source code in the gzipped tarball stored in upstream/ to make the package installable and runnable with the stack.  If this is necessary, it is done using the Linux patch command, which applies diffs to source code files.  For each change that needs to be made to the source code, generate a patch file using the command

git diff -u originalFile correctedFile > someFileName.patch

Save the someFileName.patch files in the patches/ directory of the repository.  eups will know to apply these patches after it unpacks the gzipped tarball in upstream/.

Note: eups expects the patches to be formatted according to the output of 'git diff', not the output of 'diff'.

Testing your distribution

Before finalizing the distribution, it is useful to be able to test that the distribution as set up does, in fact, build.  This can be accomplished using the lsstsw build tool.  Detailed documentation of the lsstsw build tool can be found here.  Broadly, the steps are

• Clone and set up the lsstsw package in its own directory using the instructions on the lsstsw documentation page pointed to above.

• In the lsstsw package, use the command

  Code Block
  ./bin/rebuild -r yourBranch yourPackage

  to build the development branch of your package (for this reason, development of a third party package distribution should be handled identically to development of LSST software; work on a development branch and merge to master only after a successful build and a review).

• Ideally, you should try this process on at least two different machines (one running OSX and one running a Linux distribution) to make sure that you did not accidentally benefit from the system environment of your test machine when building.

Distributing your package

Once the package builds and passes review (or vice-versa), you need to tell eups that it is available for distribution to the wide world.  To do this, add an annotated tag to your package repository using

git tag -a versionNumber -m "some comment"

The initial versionNumber should match the external package's version number.  If changes are required to the packaging (in the ups or patches directories) but not the external package source (in the upstream directory), the string ".lsst1" (and ".lsst2" etc. thereafter) should be appended to the external package's version number.  Merge your changes to master, then push your changes to the remote repository.  Push your tags to the remote repository using

git push --tags

Now you must log onto lsst-dev as the user lsstsw (this will require special permissions, so someone may need to do this for you; or you can ask Frossie to give you permission to log on as lsstsw).  Once on as lsstsw, the steps are:

• build your package with the command

  Code Block
  rebuild yourPackage

  This will cause lsst-dev to build your package and all of its dependencies.  This build will be assigned a build number formatted as bNNN.

• Once the build is complete, release it to the world using

  Code Block
  publish -b bNNN yourPackage

  This will make your package installable using

  Code Block
  eups distrib install yourPackage versionNumber

  If you wish to add a distribution server tag to your package, you can do so by changing the publish command to

  Code Block
  publish -b bNNN -t yourTag yourPackage

  Do not use the tag 'current' as that will overwrite all other packages marked as current and break the stack.  Let the people in charge of official releases handle marking things as 'current.'  it is not usually necessary to distribution-server-tag a particular third party package.

• Generally, if you're publishing a third party package, it should be because it is a dependency in the build of some (or all) top-level package(s). When the top-level package(s) are next published (and optionally tagged), your new package will be incorporated.  If you need something sooner, you can do this publishing yourself using the steps above with the top-level package. In this case, a distribution-server-tag (something like qserv-dev) is usually desirable.  That makes the top-level product (or any of its dependency components, including your third-party package) installable using

  Code Block
  eups distrib install -t yourTag packageName

Updating your external package

(NOTE: these instructions are still under construction.)

To update the version of your external package after a new upstream release, starting with a copy of the lsst stack installed the standard way, ... (more here about the initial setup requirements).

...

• eupspkg -er -v 1  prep

• eupspkg -er -v 1 config

• eupspkg -er -v 1 build
• eupspkg -er -v 1 install
• eupspkg -er -v 1 decl
• "eups list yourPackage" should now show a new version named tickets/DM-NNNN-gBLAHBLAH, where gBLAHBLAH is the git hash revision of the package.
• setup lsst_apps -t YOURTAG
• setup yourPackage tickets/DM-NNNN-gBLAHBLAH
• Run your tests.

...