Date

Location

Browser

Room System

Phone Dial-in

https://bluejeans.com/690803707

or 

https://ls.st/rsv


  1. Dial: 199.48.152.152 or bjn.vc
  2. Enter Meeting ID: 690803707 -or- use the pairing code

Dial-in numbers:

  • +1 408 740 7256
  • +1 888 240 2560 (US Toll Free)
  • +1 408 317 9253 (Alternate Number)

Meeting ID: 690803707

Attendees

Regrets

Metric Tracking Dashboard

URL:

Discussion items

ItemWhoPre-meeting notesNotes and  Action Items 
News, announcements and task review
  • The Director’s Review in preparations of the Joint Status Review is planned for 7-9 Sep 2021. There will be a strong focus on SITCom. We should present performance metric verification status. 
    • chronograf dashboard
    • pipeline Characterization Metric Report
    • example per-detector metric (e.g., 5 sigma depth)
    • documentation
  • International in-kind contributor Nacho Sevilla is interested in starting to get prepared for commissioning SV related work. What is our approach to starting to incorporate such in-kind contributions?
    • What do we need? What do we want?
    • Concept to do direct imaging survey with AuxTel
    • Getting faro to a stable place
    • What is the usage in 6 month time
    • Capability to defining work packages. Specific dataset and specific tests to run and come back with an analysis.
      • Example: is stellar locus stable, e.g., in HSC RC2? Can you make sense of the results in science context.
  • Plan is to use labels for sandbox activities
  • Postpone discussion of failure modes
  • Cronograf dashboard now set up to show metrics computed with faro for both DC2 and HSC RC2. Can select between native Gen3 processing and Gen2→Gen3 conversion. This is good milestone.
  • Nightly dashboard. There is one successful run on 11 August? Thought we disabled this job?
    • validate_drp_gen3. It was breaking. Returning all Nans. Replace with RC2 subset? Is there value in having another small dataset regularly processed, e.g., ci_imsim.
    • We used to do CFHT, DECam, and HSC running nightly.
    • According to jenkins, still running on validate_drp_gen3 nightly. jenkins is happy, but NaN values are not being displayed in cronograf.
  • Just got faro running as part of ci_imsim
  • About to deploy checking biases on the mountain. Robert will put ticket number in the minutes. We will have notebooks associated with CP verify package. Many calibration products. Deliver as notebook. Idea that this will be run every afternoon/night to check that that calibrations products were generated.

Bugs, issues, topics, etc of the week

All
  • Congratulations to Erik Dennihy who has completed his first faro ticket 
  • How do we enable faster access to large numbers of metrics in data repos (e.g., tens of thousands of per-detector metric values could be generated in a single night)?
  • Some errors were introduced to faro in  DM-31381 - Getting issue details... STATUS . See Slack threads here and here
    • Homogenization of run methods is high-priority item DM-31061 - Getting issue details... STATUS
  • What guidance do we provide for developers regarding executing faro on real data (and ensuring success) before merging changes? We need to make sure we aren't merging breaking changes – these breakages are often not evident from scons/Jenkins. (see conversation in  DM-31382 - Getting issue details... STATUS )
    • It seems we need an efficient way to actually run a pipeline against actual data repository to test the pipeline yaml files and runQuantum methods across multiple analysis contexts
  • We will need a technical solution to access large numbers of metric values. Summary task can write a aggregate statistic (e.g., mean), but could also create astropy table for example. Another solution would be putting results in relational database (but this functionality is not planned to be soon, no concrete plan to implement, not going to happen in next 3 months). 
    • Keith to repeat test with more recent weekly
    • We might need something ad hoc. Might need to compare with metadata.
    • Colin has this as high priority
  • Keith Bechtol to report on timing tests of loading thousands of metric measurements into memory with more recently weeklies   
  • Dan has PR as short term solution to consistency of run methods
  • Integration test. Executing faro on real.
    • Need small CI dataset for faro. Execute faro against some dataset.
    • Ideally, we would start from an already processed dataset
    • CI HSC has coadd outputs. 2-3 hours.
    • Is there a processed ci_hsc dataset. We could set up a cron job?
  • Multiple potential solutions to have faro run as part of routine pipeline testing and avoid causing breakages on master, i.e., make sure that jenkins run fails if faro is broken
    • Could faro pipelines be merged with DRP pipelines? (obs_lsst/pipelines/imsim/DRP.yaml)
    • Can we just add faro to ci_hsc pipelines?
  • It is possible to run specific analysis contexts of faro (e.g., per detector, per tract) separately
    • Hsin-Fang suggests that we separate out according to the stage of analysis to make more modular and possible to scale to larger datasets
  • Adding faro to ci_imsim and ci_hsc would make it so that faro is executed as part of jenkins

Reprocessing status and metrics review  

(Hsin-Fang will join the meeting at 11:30) 
Development status 
  • Fall 2021 epic  DM-30748 - Getting issue details... STATUS
  • Backlog epic:  DM-29525 - Getting issue details... STATUS

AOB

Next meeting  



List of tasks (Confluence)

DescriptionDue dateAssigneeTask appears on
  • Leanne Guy to talk to Science Pipelines (Yusra) about when do this transfer  
19 Oct 2021Leanne Guy2021-09-28 Science Metric Development Agenda and Meeting notes
  • Leanne Guy arrange to disucss at a future meeting if there are metrics from PDR3 & this paper that we might want to include in faro.   
26 Oct 2021Leanne Guy2021-08-31 Science Metric Development Agenda and Meeting notes
  • Colin to ask about capturing ideas for improvement to the stellar locus algorithm   
30 Nov 2021 2021-11-09 Science Metric Development Agenda and Meeting notes
  • Jeffrey Carlin to review metric specification package organization and the relationship to formal requirements documents
Jeffrey Carlin2022-04-19 Science Metric Development Agenda and Meeting notes
  • Colin Slater to make a preliminary draft agenda for a workshop to clarify visualization use cases for science verification and validation
Colin Slater2022-04-19 Science Metric Development Agenda and Meeting notes
  • Keith Bechtol Schedule a time to have focused discussion on verification package, potentially next status meeting
Keith Bechtol2021-09-14 Science Metric Development Agenda and Meeting notes
  • Keith Bechtol to make a ticket to better understand mapping of these camera and calibration products characterization efforts to verification documents and the focus of these efforts. Discuss with the SCLT
Keith Bechtol2021-09-14 Science Metric Development Agenda and Meeting notes