Skip to end of metadata
Go to start of metadata

(warning) This page is out of date and not actively maintained.

This page exists to coordinate DM efforts to transfer and make available (through Butler data repositories) key engineering test datasets during LSST construction. This material covers data being regularly generated by LSST-associated instrumentation (the camera, its associated test stands, the auxiliary telescope spectrograph, etc) and which is continuously ingested and made available by DM; one-off or infrequently updated precursor test datasets are covered by the regular dataset management policy.

If you are currently manually transferring data to the LDF from one of the projects listed below, please stop and coordinate your efforts with the responsible individual listed.

If you need to ensure that data from a construction activity not listed below is regularly collected and ingested, please file a JIRA Ticket (Assignee: Michelle Butler)

1.1. Data Users

See Developer Guide Storage Resources for descriptions of Butler repository locations.   These are typically in /datasets and /project/shared.   They include extra data beyond what is stored in Production that is useful for pipeline development.

1.2. Technical Details of Data Maintained by LDF

(See https://community.lsst.org/t/production-location-for-teststand-data/3775 for first announcement of production filesystem)

Dataset DescriptionOriginDestination (path to repository on GPFS)Start DateAimed for LatencyResponsible IndividualComments
Camera test stand dataSLAC
/datasets/...?


Automated rsync of BOT raw files: /lsstdata/offline/teststand/BOT

  • storage: copy of raw file

  • gen2repo: Gen2 Butler repository for files in storage 
  • log: transfer + ingestion output


User Gen2 repo location:
/project/shared/BOT

Aiming for end-September for the transfer and ingestion (available via G2 Butler) the bulk existing data. Automated transfer service started transferring BOT raw files to temporary NCSA location 03/18/2019, but no Gen2 repo yet.

~24 hours

(Automated raw file transfers and Butler ingestion < 1 hr)

Unknown User (gruendl)

(Automated raw file transfers - Greg Daues and Eric Morganson )

ETU1/2 test images taken at SLAC. Plan is to bulk transfer and ingest data from 3 repositories: SLAC test-stand,  BNL test-stand and vendor data. Estimated uncompressed total volume 60-170 TB. Additional data from other sources may be transferred if deemed useful. ~200TB estimate for total. The camera team at SLAC are reviewing to see what can be deleted before transfer to NCSA.

Images produced by ongoing test campaigns at SLAC will eventually be transferred to NCSA using the same service as for the AuxTel spectrograph, once demonstrated and functioning as a reliable service for AuxTel data.

AuxTel spectrograph test data from L1 ArchiverSummit
/lsstdata/offline/teststand/auxTel/L1Archiver
  • storage: copy of the raw file
  • gen2repo: Gen2 repository of the raw files in storage
  • log: transfer + ingestion output

User Gen2 repo location:  
/project/shared/auxTel

Email was sent on 10/17/2018 to Patrick for first non-NCSA user test.

Automated rysnc enabled to a temp NCSA location 03/20/2019

rsync of current days data starts every 10 minutes.

(Auto double check of yesterday's data happen at 00:10 and 23:10)

~15 mins from arrival at NCSA (depending upon load)


Only getting 1M/sec transfer rates. So single AuxTel file takes approx 73 seconds to transfer.

Single CCD, ~40MB spectrograph images taken during testing in Tucson and during commissioning. Aimed for (upper estimate) data rate of ~1100 images/day (50 biases, 50 darks, 50 flats, 2 images per minute for 8 hrs), ~1-2 TB/month. More likely rate expected to be 100-150 images/day, ~200GB/month. Rate expected to increase as commissioning proceeds.

The process for saving files from the test platform is different than the process that will exist during operations. On the test platform, someone chooses what files need to be saved in the permanent record of the survey. They run a program to copy the files to NCSA where a process there will put them in the correct location and ingest them into the Data Backbone.

Archive includes pre-20200101 data from Tucson and post 20200101 data from the summit.

AuxTel spectograph test images from DAQSummit/lsstdata/offline/teststand/auxTel/DAQ
(2019-05-30  visible on lsst-dev cluster but NOT LSP yet)
  • storage: copy of the export DAQ filesystem /daq-data/ats/<date> directories and mcm/<date> 
  • gen2repo: TBD Gen2 repository of the raw files in storage
  • log: transfer + ingestion output

User Gen2 repo location:
/project/shared/auxTel-ACCS

Automated rsync enabled 2019-05-30.   Automated gen2repo TBDrsync of current days data starts every 15 minutes, broken into sets of 10 files. (Auto double check of yesterday's data happen at 00:15 and 23:15)

Temporary copy of images written by the AuxTel camera control system to enable use of computing resources at NCSA.   In the future, it is expected that the actual LSST images written by the L1 Archiver will suffice.

Archive includes pre-20200101 data from Tucson and post 20200101 data from the summit.

Comcam CCS DataTucson

/lsstdata/offline/teststand/comcam/CCS/

  • storage: copy of the raw file
  • gen2repo: Gen2 repository of the raw files in storage
  • log: transfer + ingestion output
Automated rsync and ingestion complete 2020-02-18.Automated file transfer hourly
  • No labels