This page is out of date and not actively maintained.
This page is out of date and not actively maintained.
This page exists to coordinate DM efforts to transfer and make available (through Butler data repositories) key engineering test datasets during LSST construction. This material covers data being regularly generated by LSST-associated instrumentation (the camera, its associated test stands, the auxiliary telescope spectrograph, etc) and which is continuously ingested and made available by DM; one-off or infrequently updated precursor test datasets are covered by the regular dataset management policy.
If you are currently manually transferring data to the LDF from one of the projects listed below, please stop and coordinate your efforts with the responsible individual listed.
If you need to ensure that data from a construction activity not listed below is regularly collected and ingested, please file a JIRA Ticket (Assignee: Michelle Butler)
See Developer Guide Storage Resources for descriptions of Butler repository locations. These are typically in /datasets and /project/shared. They include extra data beyond what is stored in Production that is useful for pipeline development.
(See https://community.lsst.org/t/production-location-for-teststand-data/3775 for first announcement of production filesystem)
| Dataset Description | Origin | Destination (path to repository on GPFS) | Start Date | Aimed for Latency | Responsible Individual | Comments |
|---|---|---|---|---|---|---|
| Camera test stand data | SLAC | /datasets/...? Automated rsync of BOT raw files: /lsstdata/offline/teststand/BOT
User Gen2 repo location: | Aiming for end-September for the transfer and ingestion (available via G2 Butler) the bulk existing data. Automated transfer service started transferring BOT raw files to temporary NCSA location 03/18/2019, but no Gen2 repo yet. | ~24 hours (Automated raw file transfers and Butler ingestion < 1 hr) | (Automated raw file transfers - Greg Daues and Unknown User (emorganson) ) | ETU1/2 test images taken at SLAC. Plan is to bulk transfer and ingest data from 3 repositories: SLAC test-stand, BNL test-stand and vendor data. Estimated uncompressed total volume 60-170 TB. Additional data from other sources may be transferred if deemed useful. ~200TB estimate for total. The camera team at SLAC are reviewing to see what can be deleted before transfer to NCSA. Images produced by ongoing test campaigns at SLAC will eventually be transferred to NCSA using the same service as for the AuxTel spectrograph, once demonstrated and functioning as a reliable service for AuxTel data. |
| AuxTel spectrograph test data from L1 Archiver | Summit | /lsstdata/offline/teststand/auxTel/L1Archiver
User Gen2 repo location: |
Automated rysnc enabled to a temp NCSA location 03/20/2019 | rsync of current days data starts every 10 minutes. (Auto double check of yesterday's data happen at 00:10 and 23:10) Only getting 1M/sec transfer rates. So single AuxTel file takes approx 73 seconds to transfer. | Single CCD, ~40MB spectrograph images taken during testing in Tucson and during commissioning. Aimed for (upper estimate) data rate of ~1100 images/day (50 biases, 50 darks, 50 flats, 2 images per minute for 8 hrs), ~1-2 TB/month. More likely rate expected to be 100-150 images/day, ~200GB/month. Rate expected to increase as commissioning proceeds. The process for saving files from the test platform is different than the process that will exist during operations. On the test platform, someone chooses what files need to be saved in the permanent record of the survey. They run a program to copy the files to NCSA where a process there will put them in the correct location and ingest them into the Data Backbone. Archive includes pre-20200101 data from Tucson and post 20200101 data from the summit. | |
| AuxTel spectograph test images from DAQ | Summit | /lsstdata/offline/teststand/auxTel/DAQ (2019-05-30 visible on lsst-dev cluster but NOT LSP yet)
User Gen2 repo location: | Automated rsync enabled 2019-05-30. Automated gen2repo TBD | rsync of current days data starts every 15 minutes, broken into sets of 10 files. (Auto double check of yesterday's data happen at 00:15 and 23:15) | Temporary copy of images written by the AuxTel camera control system to enable use of computing resources at NCSA. In the future, it is expected that the actual LSST images written by the L1 Archiver will suffice. Archive includes pre-20200101 data from Tucson and post 20200101 data from the summit. | |
| Comcam CCS Data | Tucson | /lsstdata/offline/teststand/comcam/CCS/
| Automated rsync and ingestion complete 2020-02-18. | Automated file transfer hourly |