Skip to end of metadata
Go to start of metadata

Information resources

Catalogs

Since the DC processing happened simultaneously in two sites (NCSA and IN2P3) the resulting MySQL catalogs ended up divided into halves (with an overlap in an interval of: +5 < R.A. < +10) in between 12 databases hosted by two MySQL servers (one per site):

hostdatabasedata [GB]num. rows
lsst-db.ncsa.illinois.edu
daues_SDRP_Stripe82_ncsa
  61
  294,188,533
daues_SDRP_dedupe_byfilter_0
 683
1,729,599,837
daues_SDRP_dedupe_byfilter_1
 702
1,902,999,739
daues_SDRP_dedupe_byfilter_2
 703
1,904,627,357
daues_SDRP_dedupe_byfilter_3
1400
3,658,881,418
daues_SDRP_dedupe_byfilter_4
 692
1,751,879,716
ccdb02.in2p3.fr
lsst_prod_DC_2013_2
  62
  312,642,627
lsst_prod_dedupe_byfilter_u
 760
2,058,168,075
lsst_prod_dedupe_byfilter_g
 780
2,114,378,412
lsst_prod_dedupe_byfilter_r
 779
2,079,018,858
lsst_prod_dedupe_byfilter_i
 781
2,117,611,454
lsst_prod_dedupe_byfilter_z
 780
2,113,759,673

NCSA database names at NCSA are based on the following filter identifier-to-name mapping:

mysql> SELECT * FROM lsst_prod_DC_2013_2.Filter;
+----------+------------+----------+--------+
| filterId | filterName | photClam | photBW |
+----------+------------+----------+--------+
|        0 | u          |        0 |      0 |
|        1 | g          |        0 |      0 |
|        2 | r          |        0 |      0 |
|        3 | i          |        0 |      0 |
|        4 | z          |        0 |      0 |
|      -99 | DD         |        0 |      0 |
+----------+------------+----------+--------+

Other notes:

  • data sizes were reported for the data stored in tables not including indexes.


Calibrated exposures

NCSA

A subset of 1104591 out of 1403385 images (161 run out of 207) was achieved on NCSA HPSS service after the DC. Greg Daues has kindly restored the images at the following location (at NCSA):

% find /nfs/scratch/daues/restore/dpool/scratch/daues/SDRP_2014_NCSA/calexp_dir/sci-results/*/ -name 'calexp-*.fits.gz' | wc -l
1104591


% du -hs /nfs/scratch/daues/restore/dpool/scratch/daues/SDRP_2014_NCSA/calexp_dir/sci-results/
15T	/nfs/scratch/daues/restore/dpool/scratch/daues/SDRP_2014_NCSA/calexp_dir/sci-results/


IN2P3

The images are stored in the iRODS server:

$ ils /lsst-fr/data/DC_2013/calexps/sci-results
/lsst-fr/data/DC_2013/calexps/sci-results:
  C- /lsst-fr/data/DC_2013/calexps/sci-results/1752
  C- /lsst-fr/data/DC_2013/calexps/sci-results/1755
  ...
  C- /lsst-fr/data/DC_2013/calexps/sci-results/7202

$ ils /lsst-fr/data/DC_2013/calexps/sci-results/1752/1/g/calexp
/lsst-fr/data/DC_2013/calexps/sci-results/1752/1/g/calexp:
  bkgd-calexp-001752-g1-0040.fits
  ...
  calexp-001752-g1-0229.fits
  calexp-001752-g1-0230.fits

The files still need to be transferred to NCSA. It's estimated to be about 40 TB of data.

Coadded images


4 Comments

  1. We need to understand what we will do for image access for SDSS, both for the coadds we did, and for single epochs.  Apparently we did not save the calexps from the 2013 reprocessing.

    1. I've just revisited my conversation with Greg Daues, and he claims that: "...The full set of calexp could not be kept but there does exist "a large subset" ~20 TB gzipped . I believe this is on tape and possibly on another file system at NCSA; let me check the inventory and how to make it available...". And he has restored those files at the following location:

      % du -hs /nfs/scratch/daues/restore/dpool/scratch/daues/SDRP_2014_NCSA/calexp_dir/sci-results/
      15T	/nfs/scratch/daues/restore/dpool/scratch/daues/SDRP_2014_NCSA/calexp_dir/sci-results/

      The number of images in the folder:: 1104591. This is a bit less than what's reported in the NCSA part of the catalog (table 

      daues_SDRP_Stripe82_ncsa.Science_Ccd_Exposure): 1403385.

      Another confusing observation is the name of the folder (SDRP_2014_NCSA vs SDRP 2013). Was there any DC in 2014?



      1. That's really good news.  I think it's OK if some of the images are missing - we'll learn how to handle "missing data" that way. 

      2. There was not a Data Challenge of this scope in 2014.