Skip to end of metadata
Go to start of metadata

(back to the list of all DM-SST meeting minutes)

Logistics

Date & Time

Date : , 09:00 - 14:00 Pacific

This precedes the DMLT vF2F (07-09 Jun 2022) 

Location

Browser

Room System

Phone Dial-in

Short : ls.st/dmsst 

Zoom: https://noirlab-edu.zoom.us/j/97839121776?pwd=K1JPeUpSMXFvSXJSa2xORGkyVk5zdz09

Zoom Meeting ID: 978 3912 1776 
Zoom Password: 512314

Dial closest IP: 162.255.36.11 (east coast) and 162.255.37.11 (west coast) then use the Zoom meeting ID 978 3912 1776  as the dialing extension.

For example: 978 3912 1776 @162.255.37.11 or: 162.255.37.11##978 3912 1776  Password: 512314

Dial-in numbers:

+1 346 248 7799 (US Toll)

+1 669 900 6833 (US Toll) Meeting ID: 936 2540 1560 International numbers available: https://gemini.zoom.us/u/adcUNrbXzS

Participants


Agenda


Time (Pacific)TopicCoordinatorPre-meeting notesRunning notes and actions 

Note taker: Colin Slater 

09:00 - 9:15Welcome, project science updates

Volunteers for moderating and taking notes?

Project science updates:

  • Forecast for ComCam on telescope is being pushed out 2-4 weeks. mid-late August 2022. not expected to affect on-sky in 2023 unless TMA verification runs over
  • FDR for pumped coolant system for 1st week Aug

  • DESC SC meeting also in 1st week of August
  • DM presentation series to PST.  
    • Yusra - Status update on pipelines:  
    • Eric - Image differencing pipelines:  (tentative)
      • subsequent talks will be scheduled based on feedback from PST

Task review 

  • GPDF: If the camera is on the telescope for so long (prior to the official start of data taking), are we going to get a "surprise" request to take data?
    • Mirror is not on the telescope. Much of the purpose is for testing refrigeration, interfaces, etc. Being used for mechanical and electrical verification.
    • Might take darks/biases.
    • Keith will look up a schedule.
  • Keith Bechtol to look up more information on schedule of camera going on telescope and when data taking would occur, including calibration data such as biases, darks  
  • Robert Lupton Work out who from DM/Commissioning/Science should try to go to DESC meeting, useful to attend in person.  
  •  Robert Lupton Ask how many people from DESC will be at PCW?  

DM presentations to PST:

    • Eric is tentatively ok with the PST talk time.
    • YA: June talk is the menu of topics for future deep-dive talks.

Multi-resolution cutouts:

  • Followed up with Francisco et al. They did more testing on (ZTF?) were able to identify space-junk-glints more effectively. Not a huge selling point, could think of other ways for DM to do this.
  • Encouraged them to think more about packaging cutouts, and we'd review further as it progresses. Waiting for a more compelling case.
  • Fink broker has a paper on the space-junk-glints (not to be confused with multi-surface reflections in the telescope). https://ui.adsabs.harvard.edu/abs/2022arXiv220205719K/abstract

Brokers aren't available for PCW, so no session there. Melissa has been communicating with them. They can all have RSP accounts when it's useful to them.

Still need to set up an RSP-capabilities meeting for broker teams with Gregory Dubois-Felsmann et al.

  • Eric Bellm will circulate a Doodle to find a time for a brokers-RSP virtual meeting in June/July  


9:15 - 10:15

slide 7:

  • Jim: Surprised that surface brightness images are better for model fluxes.
    • Eli: it's possible to do model fluxes with fluence images, but you'll need to make sure to include all the right mtah.
  • RHL: Are we going to assume gray clouds?
    • Have been so far.
  • RHL: have to worry about what SED you use for the flat field
    • SED of the sky is different from SED of the sources.
  • LPG: what are the pipelines using right now for images?
    • Jim: Calexps are ("mostly") surface brightness images, because that's what you get from flat fielding. Coadds are fluence images. Should be converting to fluence images much earlier, right after doing background subtraction, and then you don't need to deal with surface brightness ever again.
    • Not currently careful enough in the pipelines about treating the difference between these images types.
  • Have to be sure we're describing this accurately to our users, including in metadata in the RSP services.


Slide 32:

  • What is the plan for doing wavelength scans with the CBP?
    • Not yet determined. Not going to be able to do a narrow wavelength scan every day. Instinct is we'll need a couple of times per year.
    • Can monitor shifts with much less CBP time, and hoping that will be sufficient.
    • Can use on-sky observations to check that the throughput isn't changing; won't tell you what changed, but would at least provide monitoring. Have to follow up to determine what happened after a throughput change.


  • CTS : how safe is it to assume that measuring dust accurately is out of scope and can it be left to DESC?
    • KB: Understanding that it is a science research project - project will adopt what the community provide
    • ER: MW dust is not a limiting factor in our calibration process
  • GPDF: need to store the weights per-source in the catalog so that recomputation is possible.
    • RHL: think it's possibly one number per source(? not sure if i'm copying this down correctly)
    • Need to settle on what this will look like, can this be an item on the Calibpalooza agenda?
    • Two topics: the data model  for preserving enough information to make the corrections possible against the released data (perhaps Calibpalooza-relevant); and the service model in the RSP for enabling users to get the corrections applied, e.g. in a light-curve query for an SN (can be worked out later).
    • Cell-based coadds may make this easier, at least for the atmospheric part. Chip-spatial-position variation may still not be constant over a cell (but Eli hopes it will be)
    • Some per-visit calibration details matter more to the SNe users, but those are not worried about the coadd

Keith: DES Y3 Appendix A has info on chromatic corrections: https://arxiv.org/pdf/2011.03407.pdf

10:15 - 10:30 Plans for CalibPaloozaLeanne and Eli are organizing a 2-day workshop on calibration strategy, affectionately named "CalibPalooza". The goal of the workshop is to define and document the full end-to-end strategy for LSST calibration, from ISR through global astrometric and photometric calibrations .  A lot of work has already been done on various aspects of calibration; what lacks is an overall vision for how it all comes together.  The outcome of the 2-days should be an initial draft of a document that defines the full LSST end-to-end calibration strategy.
  • Want to avoid spending all the time on intro material, and focus more on answering open questions.
  • At SLAC, looking like ~end of August.
  • Keith: is this thinking about "Calibration at year 1" as the goal, or year 5, etc?
    • Eli: End of Construction, for DRP.
10:30 - 11:00

Moderator: 

Eric Bellm 

Note taker: Leanne Guy 
11:00 - 11:45Completion of DAX/RSP construction capabilities

What does it take to define DAX development as complete . What remains to be done , what is the verification process 

Missing functionality : Traceability of missing DAX functionality

  • Qserv deployed for DP0.2, scaling tests continuing ...confidence in meeting goals 
  • Images services just deployed in previous weeks - needed geometrics query capbility development in Qserv - new functionality in Qserv
  • LPG: What is the problem with implementing geometric queries in Qserv
  • GPDF: Underlying mariaDB does not support geometric queries 
  • CTS: Qserv does support geometry in the query and the data are points. 
  • GPDF: Point in catalog contained in user supplied shape/query. Do not support shape in shape intersects shape queries. Has impacts mostly on image MD type queries 
  • Query history service : 2 possible designs 1) extension to TAP standard, 2) table of queries that is queryable
  • LPG: Query history service is in Gaia archive - very useful. Should this requirement in LSST be higher priority?
  • GPDF: User databases   - story was to have a large conventional DB service (Postgres) that contains a lot of system tables and MD about production processing (not in Qserv). Could be accessed via API.
  • JB:Pushing up to Qserv to share with users? PG DB also shareable, pushing not only way 
  • JB: if there is transactional user DB space, would really like to put butler space there 
  • CTS: Consolidated DB was an NCSA deliverable - who now? 
  • GPDF: USDF - transformed EFD, butler registry also there. 
  • RHL: Audit on all the DBs in project and query patterns, time series DB, visit based DB, Butler, scheduler DB as well ....
  • GPDF: Still questions about the hybrid model. Consolidated DB - the idea was to have one DB in addition to Qserv for all other DBs - common infrastructure
  • JB: Identify those that have user owned DBs - permissions, authentication, etc. 
  • GPDF: Summit system as well as USDF. 
  • Robert Lupton Review the inventory of DBs that we need and think about how we will get these deployed. Should be raised with DMLT  
  • VOSpace - looking at alternatives to WebDav, conencted ot user butler (user images, custom coadds) 
  • Recall discussion on DOIs for datasets in addition to queries. 10millions row table presents resource management issues 
  • Leanne Guy to take the question of what to do about not scoped functionality (e.g DOI) to ops-exec.    Raised with Bob 2022-06-06 – Ops will put together a plan. There aer ideas for a DM-SST -like entity in operations who could be the guardians of requirements inherited from construction (either missing scope or scoep that did not get implemented)

LPG: DAX must be finshed construction by end FY23

CTS: Need to show we have met the requirements. Clarify what is DAX, what is services 

LPG: Special Progams DB - will be next to the main DBs in Qserv. GPDF - lack of clarity on scope and scale of special programs DBs

CTS: UG databases is the biggest missing piece and main focus on development activites until end FY23

CTS: Will claim what we can on DP0.2 behaviour. APDB/PPDB are user facing but requirements are lacking . No checkbox for APDB/PPPDB

EB: implicit requirement on scheduler to provide notiifcation 2hrs in advance - testing has tested loading the history successfully, but requirements are lacking.

CTS: Add PPDB into Database-Consolidating Working Group effort.

GPDB: Why are PPs not in Qserv (not transactonal) - once per night/day is a heavy lift. Has it been absolutely excluded though? 

GPDF: Backlinks to PPDB to Object table - 

EB: Narrow table -local copy that does not live in Qserv. 

GPDF: What fraction of the Object Table would be sucked in after 1 yr of PP? have TAP in front of the PPDB  - would work but not as efficient as a DB table join. 

11:45 - 12:30Plans for compressed PVIsGregory Dubois-Felsmann What do we still need to know?

Haven't concluded whether the compressed PVIs will be usable for downstream processing, or whether users who want to do e.g. custom coadds will need to rely on the PVI-recreation service to get uncompressed coadds. That affects the load we expect the recreation service to have to support.

  • JB: Applied a criteria that we must prove (not just assert) that coadds from compressed images are safe, but there's sufficient experience DES showing that it was successful for them. So maybe we should just go ahead and make compression the baseline, rather than waiting on proof of safety.
  • Eli: Images are read and then written multiple times during single frame processing, not compatible with the need to only do lossy compression once.
    • Jim: Most of that is either changing the background or changing the mask plane, and those are different problems than modifying the compressed pixels.
  • RHL: unlikely to be accepted by the project without a sufficiently convincing study.
    • Substantial work to organize the pipelines for a single instance of compression.
    • DES order of operations was different, did no additional background subtraction after their "final cut" compressed image stage.
    • Other order-of-operations problems (from a calibration standpoint) exist in the pipelines, will need refactoring anyways.
  • GPDF: need to know what the new baseline would be, so filling out the story on e.g. bg subtraction is necessary.
  • JB: the bg subtraction changes are necessary anyways.
    • Must make sure that the standard is not "must cause no change to any science cases ever".
  • Eli: should make this a constraint on the calibration refactor; we should do the refactor such that this becomes possible, but compression not be the goal itself.


  • AP baseline includes a gap between images expiring out of the 30 day cache and them becoming available via the data release.
  • Eric: If the budget required us to pick, would assert that the trailing-year AP single visit images will be more heavily used than the DRP single visit images.
    • Eli Rykoff Report on what information DES used for their decision on using compression?  
*12:30 - 13:00 

Moderator:  Leanne Guy 

Note taker:  Colin Slater 

13:00 - 13:30Plans for the AFShttps://dmtn-226.lsst.io/v/DM-34292/index.html
  • CTS: 2 yrs into the survey - what is the distinction on ANTARES strategy vs any other broker - ignoring the construciton history
  • EB: A service woulf have a special status, prominently linked and suppoerted by Rubin, connection with CET, long term financial entanglement. Joint operational umbrella - we will make sure ANTARES keep running
  • CTS: Do we need to financially support SNAPs as well?
  • EB: Smaller team, not servicing a large community
  • MLG: ANTARES/AFS uses cases are a bit different, users might write different filters each night, change, not as easy with ANTARES, are we asking for more functionality?
  • EB: No requirement to drop a filter in on the fly  - certainly desireable. ZTF filters do  not change nightly. ANTARES have a lot of sample data to run on. User deployable filters that execute custom code is hard, happy to hand off. AFS requirements are lax, 20 simultaneous users. 
  • CTS: Hace the SAC seen this? No. Will there be an issue with perceived favourtism?
  • EB: We left this until after the selection and all brokers were selected. ANTARES and Rubin are part of NOIRLab
  • LPG: transfer of money between 2 projects within one organization - consolidation of efforts.  
  • Eric Bellm Merge PR and issue DMTN - deliver to ANTARES. Take to SAC after iteration with ANTARES.  
13:30 - 14:00 Plans for measuring non-standard alerts (DMS-REQ-270)https://dmtn-228.lsst.io/v/DM-35100/index.html

GPDF: are we confident we can achieve 5sigma as the threshold? 

EB: That is the baseline, only reason we'd use a higher threshold is if we coudn't handle 5sigma

EB: DPDD - does explicitly say to alert on not less than 10% of detections below threshold even if DMSR is not explicit. 2 separate use cases. 

JB: Request from SL for early access to postage stamps - triggerd us to come back to this. This was a way to use existing requirements to cover this request. 

MLG: Zero-flux alerts at these positions to send out the postage stamp.

EB: We will not run their custom code so they need the pixels and alerts is the only way to get the data. Wouldn't do this if they could get to images some other way. 

JB: Volumes order 100s → 1000s

GPDF: Targets need to be exposed publicly  - make sure the SLSC understands they can get this service but only by disclosing their targets. 

14:00 - 14:15Closeout & task review

14:15Close

DescriptionDue dateAssigneeTask appears on
  • Eric Bellm Create RFC for three-stamp alerts, to communicate and garner feedback  
31 Dec 2021Eric Bellm2021-11-15 DM-SST Agenda and Meeting notes
  • Robert Lupton Clarify the meaning of time in the object table. 1 sentence description in sdm_schemas, can link to a short DMTN.  Update 2022-02-09: Meeting to resolve this on 2022-02-21  
28 Feb 2022Robert Lupton2018-11-05 DM SST F2F Agenda and Meeting notes
  • Robert Lupton and Leanne Guy — review all columns in the Source & DIASource table and decide whether to remove or move to a different table. The decisions on justification should be recorded in these tables. Proposed changes will be reviewed at a future SST meeting before going to LCR. Update 2022-02-09: Meeting to resolve this on 2022-02-21  
28 Feb 2022Robert Lupton2018-11-05 DM SST F2F Agenda and Meeting notes
  • Gregory Dubois-Felsmann Submit an RFC for the compressed-PVI requirements. Make clear that the requirement on quality may require further discussion can be postponed, and should not delay moving forwards on the 6 functional requirements, which need to be approved soon.   
27 May 2022Gregory Dubois-Felsmann2021-01-11 DM-SST Agenda and Meeting notes