Database schema of catalogs (LSST GutHub)
Since the DC processing happened simultaneously in two sites (NCSA and IN2P3) the resulting MySQL catalogs ended up divided into halves (with an overlap in an interval of: +5 < R.A. < +10) in between 12 databases hosted by two MySQL servers (one per site):
host | database | data [GB] | num. rows |
---|---|---|---|
lsst-db.ncsa.illinois.edu | daues_SDRP_Stripe82_ncsa | 61 | 294,188,533 |
daues_SDRP_dedupe_byfilter_0 | 683 | 1,729,599,837 | |
daues_SDRP_dedupe_byfilter_1 | 702 | 1,902,999,739 | |
daues_SDRP_dedupe_byfilter_2 | 703 | 1,904,627,357 | |
daues_SDRP_dedupe_byfilter_3 | 1400 | 3,658,881,418 | |
daues_SDRP_dedupe_byfilter_4 | 692 | 1,751,879,716 | |
ccdb02.in2p3.fr | lsst_prod_DC_2013_2 | 62 | 312,642,627 |
lsst_prod_dedupe_byfilter_u | 760 | 2,058,168,075 | |
lsst_prod_dedupe_byfilter_g | 780 | 2,114,378,412 | |
lsst_prod_dedupe_byfilter_r | 779 | 2,079,018,858 | |
lsst_prod_dedupe_byfilter_i | 781 | 2,117,611,454 | |
lsst_prod_dedupe_byfilter_z | 780 | 2,113,759,673 |
NCSA database names at NCSA are based on the following filter identifier-to-name mapping:
mysql> SELECT * FROM lsst_prod_DC_2013_2.Filter; +----------+------------+----------+--------+ | filterId | filterName | photClam | photBW | +----------+------------+----------+--------+ | 0 | u | 0 | 0 | | 1 | g | 0 | 0 | | 2 | r | 0 | 0 | | 3 | i | 0 | 0 | | 4 | z | 0 | 0 | | -99 | DD | 0 | 0 | +----------+------------+----------+--------+ |
Other notes:
A subset of 1104591 out of 1403385 images (161 run out of 207) was achieved on NCSA HPSS service after the DC. Greg Daues has kindly restored the images at the following location (at NCSA):
% find /nfs/scratch/daues/restore/dpool/scratch/daues/SDRP_2014_NCSA/calexp_dir/sci-results/*/ -name 'calexp-*.fits.gz' | wc -l 1104591 % du -hs /nfs/scratch/daues/restore/dpool/scratch/daues/SDRP_2014_NCSA/calexp_dir/sci-results/ 15T /nfs/scratch/daues/restore/dpool/scratch/daues/SDRP_2014_NCSA/calexp_dir/sci-results/ |
The images are stored in the iRODS server:
$ ils /lsst-fr/data/DC_2013/calexps/sci-results /lsst-fr/data/DC_2013/calexps/sci-results: C- /lsst-fr/data/DC_2013/calexps/sci-results/1752 C- /lsst-fr/data/DC_2013/calexps/sci-results/1755 ... C- /lsst-fr/data/DC_2013/calexps/sci-results/7202 $ ils /lsst-fr/data/DC_2013/calexps/sci-results/1752/1/g/calexp /lsst-fr/data/DC_2013/calexps/sci-results/1752/1/g/calexp: bkgd-calexp-001752-g1-0040.fits ... calexp-001752-g1-0229.fits calexp-001752-g1-0230.fits |
The files still need to be transferred to NCSA. It's estimated to be about 40 TB of data.