2022-05-09 DM-SST Agenda and Meeting Notes

Leanne Guy

• LSST@Europe dates now confirmed for 24-28 October 2022. This will be a hybrid meeting
• PST meeting on 27 April was a science collaboration chairs presentation on the status of the Science Collaborations 2017-2022. Federica is stepping down and Wil Clarkson is taking over as chair.  Slides are here
• No PST meeting last week due to AMCR
• Eli Rykoff : RM-1120 - Getting issue details... STATUS

Gregory Dubois-Felsmann 

Current status:  Baseline is to not retain PVIs on disk in the non-lossy compressed (original) format, neither in DRP, nor AP. Concerns about the latency of providing access to single images drive the decision was to provide access to lossy-compressed images as a fallback solution. What was not made clear was whether keeping lossy-compressed versions of the images applies to DRP and AP. Previous SST discussion concluded that DRP would produce lossy-compressed PVIs and in AP they would be deleted after the 30day period. This means the driving use case (to let people get a full history of the single epoch images for a source) will not be satisfied and also that the needed  disk space needed has not been budgeted. Additional story for past 10 or so years is that we will facilite the recreation of non-lossy compressed PVIs on demand. No current prototype, no understanding of how performant such a system would be, where resources would come from, this is still-pending work.  

• Non-lossy compressed PVIs will only be avaiable for 30days from AP and never from DRP.
• Lossy compressed PVIs will be created and be avaiable as part of DRP and kept on disk for low latency access - not in DP0.2 and not done at scale. 
• At a DRP release,  ~10 months after the cut-off data, the newest image is already 10 months old. To cover this gap, AP needs to keep up to 10 months of images. The AP 30 day storage will not cover this - there will be a gap!
• Difference images are also deleted after 30days, never any proposal to compress them.
• Has been suggested that we ask brokers to take on some of this and ask them to archive 1-2 years of PVIs (to account for the latency of generating DRPs) 
• AP is meant to enable prompt follow up, if the goal is to do science in that gap, then the guidance is to wait for a DRP. Is this 10month gap coverage needed for prompt followup or is it 'to do science'? What are the use cases involving followup?
  • strong lensing - longer than 30 days - can request and get it
  • AGN 
  • Solar System need to go back and stack images to find precursor events  lower than the SNR 
• Certan use cases where we have to wait fo the DRP, AP cannot do everthing - diff. processing not because of saving money on disk cost.
• Need an estimate of how much it will cost to store vs how much to recreate on the sky. If there is high demand on a recreation system with no quota, it's better to store everything. 
•  Lossy compressed WG did not actually quantiy the impact on a particular science case, nor if a good job of coadding can be done on a compressed image. 
• DES used lossy compressed images, there were technical problems withm with fpack (now fixed) but compressing the variance plane did not work correctly because the algorithm. Error bars on photometric & astrometric calibrations are way bigger than compression. DES show it is reasonable and can do do a decent job.
• For repeatability we need consistency in  processing.  

• Gregory Dubois-Felsmann  / Leanne Guy Create a TN that outlines the science use cases and options for compressed PVIs from AP. Present the issue at DMLT VF2F and then the PST.  13 Jun 2022 

Jeffrey Carlin 

Leanne Guy

Next meeting is: 23 May 2022 at 11:00 Pacific