Forecast for ComCam on telescope is being pushed out 2-4 weeks. mid-late August 2022. not expected to affect on-sky in 2023 unless TMA verification runs over
FDR for pumped coolant system for 1st week Aug
DESC SC meeting also in 1st week of August
DM presentation series to PST.
Yusra - Status update on pipelines:
Eric - Image differencing pipelines: (tentative)
subsequent talks will be scheduled based on feedback from PST
Task review
GPDF: If the camera is on the telescope for so long (prior to the official start of data taking), are we going to get a "surprise" request to take data?
Mirror is not on the telescope. Much of the purpose is for testing refrigeration, interfaces, etc. Being used for mechanical and electrical verification.
Might take darks/biases.
Keith will look up a schedule.
Keith Bechtol to look up more information on schedule of camera going on telescope and when data taking would occur, including calibration data such as biases, darks
Robert Lupton Work out who from DM/Commissioning/Science should try to go to DESC meeting, useful to attend in person.
Robert Lupton Ask how many people from DESC will be at PCW?
DM presentations to PST:
Eric is tentatively ok with the PST talk time.
YA: June talk is the menu of topics for future deep-dive talks.
Followed up with Francisco et al. They did more testing on (ZTF?) were able to identify space-junk-glints more effectively. Not a huge selling point, could think of other ways for DM to do this.
Encouraged them to think more about packaging cutouts, and we'd review further as it progresses. Waiting for a more compelling case.
Brokers aren't available for PCW, so no session there. Melissa has been communicating with them. They can all have RSP accounts when it's useful to them.
Still need to set up an RSP-capabilities meeting for broker teams with Gregory Dubois-Felsmann et al.
Eric Bellm will circulate a Doodle to find a time for a brokers-RSP virtual meeting in June/July
Jim: Surprised that surface brightness images are better for model fluxes.
Eli: it's possible to do model fluxes with fluence images, but you'll need to make sure to include all the right mtah.
RHL: Are we going to assume gray clouds?
Have been so far.
RHL: have to worry about what SED you use for the flat field
SED of the sky is different from SED of the sources.
LPG: what are the pipelines using right now for images?
Jim: Calexps are ("mostly") surface brightness images, because that's what you get from flat fielding. Coadds are fluence images. Should be converting to fluence images much earlier, right after doing background subtraction, and then you don't need to deal with surface brightness ever again.
Not currently careful enough in the pipelines about treating the difference between these images types.
Have to be sure we're describing this accurately to our users, including in metadata in the RSP services.
Slide 32:
What is the plan for doing wavelength scans with the CBP?
Not yet determined. Not going to be able to do a narrow wavelength scan every day. Instinct is we'll need a couple of times per year.
Can monitor shifts with much less CBP time, and hoping that will be sufficient.
Can use on-sky observations to check that the throughput isn't changing; won't tell you what changed, but would at least provide monitoring. Have to follow up to determine what happened after a throughput change.
CTS : how safe is it to assume that measuring dust accurately is out of scope and can it be left to DESC?
KB: Understanding that it is a science research project - project will adopt what the community provide
ER: MW dust is not a limiting factor in our calibration process
GPDF: need to store the weights per-source in the catalog so that recomputation is possible.
RHL: think it's possibly one number per source(? not sure if i'm copying this down correctly)
Need to settle on what this will look like, can this be an item on the Calibpalooza agenda?
Two topics: the data model for preserving enough information to make the corrections possible against the released data (perhaps Calibpalooza-relevant); and the service model in the RSP for enabling users to get the corrections applied, e.g. in a light-curve query for an SN (can be worked out later).
Cell-based coadds may make this easier, at least for the atmospheric part. Chip-spatial-position variation may still not be constant over a cell (but Eli hopes it will be)
Some per-visit calibration details matter more to the SNe users, but those are not worried about the coadd
Leanne GuyGregory Dubois-FelsmannEli Rykoff Meet to understand what user-facing interface aspects need to be planned in order to support the per-source SED corrections in catalogs.
Leanne and Eli are organizing a 2-day workshop on calibration strategy, affectionately named "CalibPalooza". The goal of the workshop is to define and document the full end-to-end strategy for LSST calibration, from ISR through global astrometric and photometric calibrations . A lot of work has already been done on various aspects of calibration; what lacks is an overall vision for how it all comes together. The outcome of the 2-days should be an initial draft of a document that defines the full LSST end-to-end calibration strategy.
Want to avoid spending all the time on intro material, and focus more on answering open questions.
At SLAC, looking like ~end of August.
Keith: is this thinking about "Calibration at year 1" as the goal, or year 5, etc?
Qserv deployed for DP0.2, scaling tests continuing ...confidence in meeting goals
Images services just deployed in previous weeks - needed geometrics query capbility development in Qserv - new functionality in Qserv
LPG: What is the problem with implementing geometric queries in Qserv
GPDF: Underlying mariaDB does not support geometric queries
CTS: Qserv does support geometry in the query and the data are points.
GPDF: Point in catalog contained in user supplied shape/query. Do not support shape in shape intersects shape queries. Has impacts mostly on image MD type queries
Query history service : 2 possible designs 1) extension to TAP standard, 2) table of queries that is queryable
LPG: Query history service is in Gaia archive - very useful. Should this requirement in LSST be higher priority?
GPDF: User databases - story was to have a large conventional DB service (Postgres) that contains a lot of system tables and MD about production processing (not in Qserv). Could be accessed via API.
JB:Pushing up to Qserv to share with users? PG DB also shareable, pushing not only way
JB: if there is transactional user DB space, would really like to put butler space there
CTS: Consolidated DB was an NCSA deliverable - who now?
GPDF: USDF - transformed EFD, butler registry also there.
RHL: Audit on all the DBs in project and query patterns, time series DB, visit based DB, Butler, scheduler DB as well ....
GPDF: Still questions about the hybrid model. Consolidated DB - the idea was to have one DB in addition to Qserv for all other DBs - common infrastructure
JB: Identify those that have user owned DBs - permissions, authentication, etc.
GPDF: Summit system as well as USDF.
Robert Lupton Review the inventory of DBs that we need and think about how we will get these deployed. Should be raised with DMLT Update 2022-10-11: Robert Lupton will handle this in the commissioning context
VOSpace - looking at alternatives to WebDav, conencted ot user butler (user images, custom coadds)
Recall discussion on DOIs for datasets in addition to queries. 10millions row table presents resource management issues
Leanne Guy to take the question of what to do about not scoped functionality (e.g DOI) to ops-exec. Raised with Bob 2022-06-06 – Ops will put together a plan. There aer ideas for a DM-SST -like entity in operations who could be the guardians of requirements inherited from construction (either missing scope or scoep that did not get implemented)
LPG: DAX must be finshed construction by end FY23
CTS: Need to show we have met the requirements. Clarify what is DAX, what is services
LPG: Special Progams DB - will be next to the main DBs in Qserv. GPDF - lack of clarity on scope and scale of special programs DBs
CTS: UG databases is the biggest missing piece and main focus on development activites until end FY23
CTS: Will claim what we can on DP0.2 behaviour. APDB/PPDB are user facing but requirements are lacking . No checkbox for APDB/PPPDB
EB: implicit requirement on scheduler to provide notiifcation 2hrs in advance - testing has tested loading the history successfully, but requirements are lacking.
CTS: Add PPDB into Database-Consolidating Working Group effort.
GPDB: Why are PPs not in Qserv (not transactonal) - once per night/day is a heavy lift. Has it been absolutely excluded though?
GPDF: Backlinks to PPDB to Object table -
EB: Narrow table -local copy that does not live in Qserv.
GPDF: What fraction of the Object Table would be sucked in after 1 yr of PP? have TAP in front of the PPDB - would work but not as efficient as a DB table join.
Haven't concluded whether the compressed PVIs will be usable for downstream processing, or whether users who want to do e.g. custom coadds will need to rely on the PVI-recreation service to get uncompressed coadds. That affects the load we expect the recreation service to have to support.
JB: Applied a criteria that we must prove (not just assert) that coadds from compressed images are safe, but there's sufficient experience DES showing that it was successful for them. So maybe we should just go ahead and make compression the baseline, rather than waiting on proof of safety.
Eli: Images are read and then written multiple times during single frame processing, not compatible with the need to only do lossy compression once.
Jim: Most of that is either changing the background or changing the mask plane, and those are different problems than modifying the compressed pixels.
RHL: unlikely to be accepted by the project without a sufficiently convincing study.
Substantial work to organize the pipelines for a single instance of compression.
DES order of operations was different, did no additional background subtraction after their "final cut" compressed image stage.
Other order-of-operations problems (from a calibration standpoint) exist in the pipelines, will need refactoring anyways.
GPDF: need to know what the new baseline would be, so filling out the story on e.g. bg subtraction is necessary.
JB: the bg subtraction changes are necessary anyways.
Must make sure that the standard is not "must cause no change to any science cases ever".
Eli: should make this a constraint on the calibration refactor; we should do the refactor such that this becomes possible, but compression not be the goal itself.
AP baseline includes a gap between images expiring out of the 30 day cache and them becoming available via the data release.
Eric: If the budget required us to pick, would assert that the trailing-year AP single visit images will be more heavily used than the DRP single visit images.
Eli Rykoff Report on what information DES used for their decision on using compression?
CTS: 2 yrs into the survey - what is the distinction on ANTARES strategy vs any other broker - ignoring the construciton history
EB: A service woulf have a special status, prominently linked and suppoerted by Rubin, connection with CET, long term financial entanglement. Joint operational umbrella - we will make sure ANTARES keep running
CTS: Do we need to financially support SNAPs as well?
EB: Smaller team, not servicing a large community
MLG: ANTARES/AFS uses cases are a bit different, users might write different filters each night, change, not as easy with ANTARES, are we asking for more functionality?
EB: No requirement to drop a filter in on the fly - certainly desireable. ZTF filters do not change nightly. ANTARES have a lot of sample data to run on. User deployable filters that execute custom code is hard, happy to hand off. AFS requirements are lax, 20 simultaneous users.
CTS: Hace the SAC seen this? No. Will there be an issue with perceived favourtism?
EB: We left this until after the selection and all brokers were selected. ANTARES and Rubin are part of NOIRLab
LPG: transfer of money between 2 projects within one organization - consolidation of efforts.
Eric Bellm Merge PR and issue DMTN - deliver to ANTARES. Take to SAC after iteration with ANTARES.
13:30 - 14:00
Plans for measuring non-standard alerts (DMS-REQ-270)
GPDF: are we confident we can achieve 5sigma as the threshold?
EB: That is the baseline, only reason we'd use a higher threshold is if we coudn't handle 5sigma
EB: DPDD - does explicitly say to alert on not less than 10% of detections below threshold even if DMSR is not explicit. 2 separate use cases.
JB: Request from SL for early access to postage stamps - triggerd us to come back to this. This was a way to use existing requirements to cover this request.
MLG: Zero-flux alerts at these positions to send out the postage stamp.
EB: We will not run their custom code so they need the pixels and alerts is the only way to get the data. Wouldn't do this if they could get to images some other way.
JB: Volumes order 100s → 1000s
GPDF: Targets need to be exposed publicly - make sure the SLSC understands they can get this service but only by disclosing their targets.
