Excellent talk from Eli at last week's PST meeting on calibration
SCOC workshop on cadence this week
Document written by Chuck, Leanne, Robert, Brian, etc on missing scope. Many entries relevant to DM/DM-SST
Rubin-Euclid DDP report. Where/Who will be serving them? Out of scope for this report, this is the next step. Rubin is not expected to produce/host/serve them as part of it's current scope.
No major reactions to Spencer's outline of Alert Distribution interfaces (auth, etc.)
Brokers continue to be very unsure about practical limitations of access to data products through the Science Platform & it's clear this is hindering their planning
Interest in bulk access to historical alerts (~tarballs) in addition to present DMTN-183 design
Discussion among brokers about potentially using a shared alert archive
Francisco Forster strongly arguing for larger cutouts–an interesting proposal to make them multi-resolution to keep the overall size (in bytes) similar. Followup experiments on ZTF confirm (moderate) improvement in image-based classification (slides).
Spencer is getting alert distribution set up at IDF right now.
Cutouts, access to PPDB. Eric pointed them to sizing model document but community still not sure how to plan.
Access to alert archive has not been confronted yet
Community concerned about volume limitations, 1, 10, more cutouts.
Cloning the PPDB, dump of the diff?
What is not working about the standard cutout service? Science platform was not designed for this. Careful about pushing the RSP into a role it was not designed for.
Also supplemental queries to what is in the packet, cutouts larger than alerts,
Should we give larger cutouts in the first place.
Collect a list of issues ahead of time to think about our responses.
Also uncertainty on our side.
Spencer has a prototype for the alert archive, focus on small searches, pick out individual alerts.
Alerce have a deep NN classifier on the ZTF stamps , variable, AGN, SN classification. FF concern is that the current cutout size prevents classification. Proposal is a hierarchical resolution - larger spatial area but same overall number of pixels.
Is this the right approach?
How do we represent the data?
How do we decide the scale?
Little concern from other brokers. Alerce is doing the most stamp processing.
GPDF: No real concerns, good experiment.
Are the coarser pixels for the central regions included? FF used annuli to avoid duplication and for size concerns.
Can pack this into a FITS file but an image viewer will not recognize it as an image.
Could also adopt a stricter lossy type compression. Most compression algorithms are not designed around small things.
How much are they looking for in the template images vs difference images.
Orig. pres is about host association → template
What about a low resolution as well?
Could we have both options?
Do brokers need to take everything? Can throw away what people want to keep. Indicated that we can remove some parts of the alerts, e.g remove the history and have just the Object/DIASource record. No team has committed to this yet.
Eric Bellm organize an informal round-table with the brokers to understand their needs.
Eric Bellm will follow up on the topic of larger/multi-resolution cutouts with Francisco F and report back in ~ 4 months
strong ensemble of community brokers lessens the need for an on-project backstop service
Required system has challenging implementation characteristics and only iffy performance
DMTN-165 hybrid alert approach remains viable & interesting but hard to prioritize it over other work
staffing challenges
Requirements to strike:
DMS-REQ-0348
DMS-REQ-0343
DMS-REQ-0342
LSR-REQ-0025 is missing? flows down to DMS-REQ-0342
LSR-REQ-0026 ?
POC on AFS done, claimed the milestone but a lot of work to do to build it.
Formally retire the risk on "no brokers"
KSK: This was the only interface that promised RT access to the alert. No broker is under any requirement to turn around the alerts in any given time. AFS was for the
EB: No latency requirement on the AFS, OTT1. We would aim to but we have no requirement.
Brokers will be motivated to make the turnaround fast. We can follow up.
AFS: Also only place where the whole DR community could put their filters.
CS: latency could be backfilled with PPDB broadly addressing some use cases - can the latency be reduced. Significant implications for the architecture.
AP would be better served focusing on algorithms and pipelines.
GPDF: Want to be careful about descoping a promised service.
ZI: put the filtering service into the SRD, there was not guarantee that any broker would be funded. If antares is the broker at the US national level, we could descope. Need some agreement btw NL, NSF, Rubin.
LCR still being drafted, to combine with above descope
change would be to DMS-REQ-0274
(is an LCR required here? Additional contents don't violate the requirements...)
Alert size is only specified in the DPDD
Eric Bellm Create RFC for three-stamp alerts, to communicate and garner feedback RFC-904
Two-stamp approach is mentioned in change-controlled design documents (LDM-151 and LDM-612) in addition to the DPDD; these would need to be changed in any event.
The alert distribution system is reaching maturity and we would like to consider handing over to ops. What do we need to demonstrate to deliver on the construction requirements? What can be passed to ops (eg P2/3 requirements).
DMS-REQ-0002 (1b)
DMS-REQ-0391 (2)
DMS-REQ-0393 (2) ?
DMS-REQ-0094 (1b) ?
Planning for an acceptance test campaign.
Prompt Processing system adjacent ot the AD system. Had nover one without handing over the other.
Can we really test and verify AD without having any online processing.
Decouple the verification of requirements from the handover to ops. Look to put together a test plan now and test on precursor datasets at the IDF. Then later look to rerun at SLAC later with LSSTCam data.
Eric BellmLeanne Guy to work on a plan to hand over the Alert Distribution System to Ops
Robert Lupton Clarify the meaning of time in the object table. 1 sentence description in sdm_schemas, can link to a short DMTN. Update 2022-02-09: Meeting to resolve this on 2022-02-21
Leanne Guy Prepare to consult the PST on the question of providing compressed PVIs for AP outputs, to cover the period before the data become available in a DR.
Parker FagreliusPatrick Ingraham how long will it take to do a scan as described? No need to scan the whole WL range but will require additional points outside nominal lambda range.
Eli Rykoff , Leanne Guy Develop a proposal for what calibration processing, hardware, data we actually need and what will be needed for DR1. This has implications for the ORR and for prioritisation of work in commissioning