Skip to end of metadata
Go to start of metadata

Preliminaries

The SDSS images from Stripe 82 need to reside on your local machine to run some of the LSST tutorials (see: Process SDSS Stripe 82 Images). 

Load the LSST Environment

You must have the LSST Stack installed on your system (see LSST Stack Installation) to proceed. The commands listed in the code blocks below primarily assume you are using the bash shell; analogous commands for (t)csh should work as well. If you have not already done so, load the LSST environment:

source $INSTALL_DIR/loadLSST.bash          # bash users

where $INSTALL_DIR is the directory where the LSST Stack was installed. 

Download Tutorials Package

If you have not already done so, download the tutorials package, which includes a utility for generating the URLs for images in the SDSS archive. Fetch the package from the LSST source code repository to a working directory, and set up an environment variable: 

cd /path/to/install/directory
git clone https://github.com/lsst-dm/tutorials.git
export TUT_DIR=$PWD/tutorials/sdssDrpTutorial

Configure the Data

An area of interest, a 0.2° x 0.2° region of SDSS Stripe 82 in the r-band, was defined in the tutorial Process SDSS Stripe 82 Images. Even this relatively limited area requires over 700 SDSS fields of input data. 

Acquire the Data

Install the Astrometry Catalog

Acquire and install the astrometry_net_data package for Stripe 82 to enable astrometric solutions (a capability not used in this example) and matching the astrometric catalog to detected sources: 

curl -O http://lsst-web.ncsa.illinois.edu/sdss-s12/sdss-2012-05-01-0.tgz 
tar xzf sdss-2012-05-01-0.tgz 
eups declare -r sdss-2012-05-01-0 astrometry_net_data sdss-2012-05-01-0
setup astrometry_net_data sdss-2012-05-01-0 --keep

Note: The astrometry_net_data package is roughly 38 MB. 

Fetch the SDSS DR7 Data

The DR7 data can be acquired from the SDSS archive at http://das.sdss.org/imaging/. The data must include the asTransfpC, fpM, psField, and tsField files, with a file structure mirroring that of the SDSS server. (See the SDSS Data Processing & Products page for data product definitions.) Starting from the file of image IDs that was generated by the refCoaddList.py script in the tutorial: 

--id camcol=3 field=11 filter=r rerun=40 run=2650
--id camcol=3 field=28 filter=r rerun=40 run=5765
--id camcol=3 field=29 filter=r rerun=40 run=5765

Use the genRetrieveList.py script in the tutorials package to generate a set of URLs to the needed data products in the SDSS archive, then use wget to both retrieve the files from the SDSS archive and preserve the native directory structure: 

cd $DEMO_DIR/Stripe82
$TUT_DIR/python/genRetrieveList.py rawInputs_r.txt retrieve.txt
wget -r -b -R "index.html*" -np -nH --cut-dirs=1 -P ./runs -i retrieve.txt

The download of these ~700 fields may take up to an hour and consume ~2.7 GB of disk space. Once retrieved, the data will be located in subdirectories like the following: 

SDSS Data Directory Tree
/stripe82
  /runs
    _mapper
    registry.sqlite3
    /1755
      /40
        /astrom
          /asTrans-001755.fit
        /calibChunks
          /<camcol>
            /tsField-001755-1-40-0409.fit
            /tsField-00<run>-<camcol>-<rerun>-<field>.fit
        /corr
          /<camcol>
            /fpC-001755-r1-0409.fit.gz
            /fpC-00<run>-<filter><camcol>-<field>.fit.gz
        /objcs
          /<camcol>
            /fpM-001755-r1-0409.fit
            /fpM-00<run>-<filter><camcol>-<field>.fit
            /psField-001755-1-0409.fit
            /psField-00<run>-<camcol>-<field>.fit
    /2649
      [...]  

Create a Registry

A registry file must be created to contain essential meta-data (filter used, sky coverage, etc.) from the input image file headers for later use. Create one and place it in your data repository:  

setup obs_sdss             # if you have not already setup this package
genInputRegistry.py runs   # default output filename is "registry.sqlite3"
mv registry.sqlite3 ./runs

Setup a Database Server

A database server, and the SeasonFieldQuality database table, is necessary for following the Summer 2013 demo (Process SDSS Stripe 82 Images) to co-add and perform forced photometry on Stripe 82 images. If you have write access to the database server at NCSA this step is unnecessary.  

Install MySQL

Scisql requires version 5 of MySQL.  (See for example: http://dev.mysql.com/downloads/mysql/  As of writing, lsst-db.ncsa.illinois.edu is running the Oracle-supported MySQL, though this may change.)  Check that you can connect to your new database server and create databases.

Install scisql

Scisql deploys a number of database UDFs for spatial indexing, spherical geometry, and common astronomical operations like converting between flux and magnitudes. To install scisql,  see https://github.com/smonkewitz/scisql or follow these steps recently tested on OS X 10.11.

export PREFIX=/usr/local/mysql  
# This is the directory with the mysql installation
# The executable for the mysql client should be in $PREFIX/bin/mysql for example  
git clone https://github.com/smonkewitz/scisql.git  # For the latest
cd scisql
git checkout tickets/DM-4867
./configure --prefix=$PREFIX --mysql-includes=$PREFIX/include
# If you get an error and you are sure you have an appropriate version, edit the tools/mysqlversion.py to always succeed and rerun configure.)
[sudo] make install

Check that the scisql libraries and deploy scripts are in $PREFIX. 

Next, deploy the UDFs. 

export PYTHONPATH:$PREFIX/python:$PYTHONPATH
sudo python $PREFIX/bin/scisql-deploy.py --mysql-dir=$PREFIX --mysql-socket=/tmp/mysql.sock

Note, to find the socket for your mysql installation, run:

mysqladmin variables -u root -p

Back in your mysql client, check that the scisql UDFs have been successfully installed:

mysql> SELECT scisql_abMagToDn(20.5, 3.0e+12);
+---------------------------------+
| scisql_abMagToDn(20.5, 3.0e+12) |
+---------------------------------+
|               18928.72033440577 |
+---------------------------------+

 

Create the SDSS_quality_db

To create a copy of the SDSS quality database locally, create a database on your server.

-- Choose a name for the new database
-- To match the name of the database on the ncsa server: 
mysql> create database krughoff_SDSS_quality_db;

Finally, download the mysqldumps and load into your local server:

curl -O http://lsst-web.ncsa.illinois.edu/~yusra/SeasonFieldQuality/krughoff_SDSS_quality_db.sql.gz
curl -O http://lsst-web.ncsa.illinois.edu/~yusra/SeasonFieldQuality/krughoff_SDSS_quality_db_htm.sql.gz
gunzip krughoff_SDSS_quality_db.sql.gz
gunzip krughoff_SDSS_quality_db_htm.sql.gz
mysql -u root -p krughoff_SDSS_quality_db < krughoff_SDSS_quality_db.sql
mysql -u root -p krughoff_SDSS_quality_db < krughoff_SDSS_quality_db_htm.sql


  • No labels