Date

 

Location

Zoom alias : https://ls.st/rsv

Zoom:  https://noirlab-edu.zoom.us/j/96120781994?pwd=bCtpT3Q3b2RsU1ZBRUFaSnUrZXo3Zz09#success


Attendees

Regrets

Metric tracking dashboards

Rubin science pipelines weekly change log and  Github repo: https://github.com/lsst-dm/lsst_git_changelog

Status of jenkins jobs and slack channle for notifications:  #dmj-s_verify_drp_metrics slack channel 

Discussion items

ItemWhoPre-meeting notesMinutes
News, announcements and task review

All



CI status, bugs, issues of the week

All
  • wPerp. This does not seem to be a multimatch issue. We need a ticket for wPerp specifically. There may be a short term fix.
    • Jeffrey Carlin to make a ticket for wPerp issue and add Yusra as watcher .
  • Separate issue with multimatch. Consumes large amount of memory. Our proposed solution is communicating that the fix is on the way. In parallel, we need to make progress on parquet conversion of the catalog matching.
    • Jeffrey Carlin to try the backport to make a minimal change to add filtering prior to catalog matching, still using the FITS catalog inputs (see DM-26987). Assess feasibility to make the backport.  
  • JIRA ticket for multimatch
  • The parquet conversion involves a few steps
    • Merge the current selectors PR
    • Improvements to catalog matching
      • Shorter term fixes:
        • Use column store
        • Filtering on sources
      • There is a longer term issue of matching algorithm. Tasks should have an outer layer that speaks butler, and an inner layer that works with in-memory objects.
    • Converting other metrics to use parquet inputs
  • Back porting to v23 processing. The deadline is in a few days.
  • We have an option to turn off faro for DP0.2 processing
Faro Development status 

Leanne Guy

  • Fall 2021 epic DM-30748 - Getting issue details... STATUS
  • Spring 2022 epic  DM-33385 - Getting issue details... STATUS
  • Backlog epic:  DM-29525 - Getting issue details... STATUS
  • Progress on conversion to parquet
  • In-progress and ToDo tickets have been moved to the new S22 epic. There are 3 remaining tickets to be closed out (currently In Review or Reviewed) in the F21 epic, then we can close out the f21 epic.
  • Status of selectors PR (DM-32611)
    • Building off of selector actions
    • Introducing base class for measurement configs to specify the columns to be loaded from the parquet tables in a more straightforward way. May want to consult with Jim whether this belongs with pexConfig before merging into faro.

Data processing campaigns  

Jeffrey Carlin Keith Bechtol

  • RC2/DC2 reprocessing epic : DM-26911 - Getting issue details... STATUS
    • w_2021_40 RC2:   DM-32248 - Getting issue details... STATUS
    • w_2021_44 DC2 : DM-32391 - Getting issue details... STATUS
  • AuxTel
  • AP/DiffIm

DP0.2

Colin Slater 

Leanne Guy  

Faro for DP0.2


AOB

Leanne Guy

Potential co-working session ideas here

Next meeting  


List of tasks (Confluence)

DescriptionDue dateAssigneeTask appears on
  • Leanne Guy to talk to Science Pipelines (Yusra) about when do this transfer  
19 Oct 2021Leanne Guy2021-09-28 Science Metric Development Agenda and Meeting notes
  • Leanne Guy arrange to disucss at a future meeting if there are metrics from PDR3 & this paper that we might want to include in faro.   
26 Oct 2021Leanne Guy2021-08-31 Science Metric Development Agenda and Meeting notes
  • Colin to ask about capturing ideas for improvement to the stellar locus algorithm   
30 Nov 2021 2021-11-09 Science Metric Development Agenda and Meeting notes
  • Colin Slater to make a preliminary draft agenda for a workshop to clarify visualization use cases for science verification and validation
 Colin Slater2022-04-19 Science Metric Development Agenda and Meeting notes
  • Jeffrey Carlin to review metric specification package organization and the relationship to formal requirements documents
 Jeffrey Carlin2022-04-19 Science Metric Development Agenda and Meeting notes
  • Keith Bechtol Schedule a time to have focused discussion on verification package, potentially next status meeting
 Keith Bechtol2021-09-14 Science Metric Development Agenda and Meeting notes
  • Keith Bechtol to make a ticket to better understand mapping of these camera and calibration products characterization efforts to verification documents and the focus of these efforts. Discuss with the SCLT
 Keith Bechtol2021-09-14 Science Metric Development Agenda and Meeting notes