Versions Compared

Old Version 14

changes.mady.by.user Michelle Gower

Saved on

compared with

New Version 15

changes.mady.by.user Michelle Gower

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Dataset DescriptionOriginDestination (path to repository on GPFS)Start DateAimed for LatencyResponsible IndividualComments
Camera test stand dataSLAC
/datasets/...?


Automated rsync of BOT raw files: /project/production/tmpdataloc/BOT

  • storage: copy of raw file

  • gen2repo: Gen2 Butler repository for files in storage (TBD: waiting for lsstCam mapper)
  • log: transfer + ingestion output
Aiming for end-September for the transfer and ingestion (available via G2 Butler) the bulk existing data. Automated transfer service started transferring BOT raw files to temporary NCSA location 03/18/2019, but no Gen2 repo yet.

~24 hours

(Automated raw file transfers and Butler ingestion < 1 hr)

Robert Gruendl

(Automated raw file transfers - Greg Daues and Michelle Gower )

ETU1/2 test images taken at SLAC. Plan is to bulk transfer and ingest data from 3 repositories: SLAC test-stand,  BNL test-stand and vendor data. Estimated uncompressed total volume 60-170 TB. Additional data from other sources may be transferred if deemed useful. ~200TB estimate for total. The camera team at SLAC are reviewing to see what can be deleted before transfer to NCSA.

Images produced by ongoing test campaigns at SLAC will eventually be transferred to NCSA using the same service as for the AuxTel spectrograph, once demonstrated and functioning as a reliable service for AuxTel data.

AuxTel spectrograph test data from L1 ArchiverTucson
/project/production/tmpdataloc/auxTel
  • storage: copy of the raw file
  • gen2repo: Gen2 repository of the raw files in storage
  • log: transfer + ingestion output

Email was sent on 10/17/2018 to Patrick for first non-NCSA user test.

Automated rysnc enabled to a temp NCSA location 03/20/2019

rsync of current days data starts every 10 minutes.

(Auto double check of yesterday's data happen at 00:10 and 23:10)

~15 mins from arrival at NCSA (depending upon load)


Only getting 1M/sec transfer rates. So single AuxTel file takes approx 73 seconds to transfer.

Michelle Gower

Single CCD, ~40MB spectrograph images taken during testing in Tucson and during commissioning. Aimed for (upper estimate) data rate of ~1100 images/day (50 biases, 50 darks, 50 flats, 2 images per minute for 8 hrs), ~1-2 TB/month. More likely rate expected to be 100-150 images/day, ~200GB/month. Rate expected to increase as commissioning proceeds.

The process for saving files from the test platform is different than the process that will exist during operations. On the test platform, someone chooses what files need to be saved in the permanent record of the survey. They run a program to copy the files to NCSA where a process there will put them in the correct location and ingest them into the Data Backbone.

AuxTel spectograph test images from DAQTucson/lsstdata/offline/teststand/auxTel/DAQ
(2019-05-30  visible on lsst-dev cluster but NOT LSP yet)
  • storage: copy of the export DAQ filesystem /daq-data/ats/<date> directories and mcm/<date> 
  • gen2repo: TBD Gen2 repository of the raw files in storage
  • log: transfer + ingestion output
Automated rsync enabled 2019-05-30.   Automated gen2repo TBDrsync of current days data starts every 15 minutes, broken into sets of 10 files. (Auto double check of yesterday's data happen at 00:15 and 23:15)Michelle GowerTemporary copy of images written by the AuxTel camera control system to enable use of computing resources at NCSA.   In the future, it is expected that the actual LSST images written by the L1 Archiver will suffice.