Page History

Leanne Guy

See notes from previous meeting

A few items that came up in the DRP metrics monitoring meeting this week:

• Is it possible to add an additional field for a timestamp when dispatching results to SQuaSH so that we can control the time axis value of results for plotting performance over time? Currently, the time is the time of upload to SQuaSH rather than something directly tied to the processing / data collection time.
• Goal to get faro people talking directly to Monika / people doing the RC2 data reduction. The desired behavior is that Monika runs the data reduction, faro is run as part of the pipeline, and metric results are uploaded to SQuaSH all automatically.

"First-look Analysis and Feedback Functionality (FAFF)" group starting up this week, beginning with a survey of use cases.

A grad student at Stanford working with Eric Charles, Sidney Mau, contacted Keith to indicate that he is continuing to develop the N-way matcher algorithm. Keith pointed him to the MultiMatch and GroupView code and where it is used in faro so that he can attempt to implement a drop-in replacement.

• Metrics shipping to influxDB. Still a manual part to get into SQuaSH, Simon reviews the tags for datasets. Will go away in future when APIs are homogenized. Visualization done with chronograph.
• Given the manual step, the timestamp work has been moved up in priority.
  • Time written to disk is the default 
  • Parameter will be added to the upload step to override this and provide a timestamp. 
• Want to put in place a single pipeline to run a reprocessing + faro + upload to squash. 

•  Jeffrey CarlinCheck with Monika that NCSA have what they need to run faro in a single pipeline for reprocessing 06 Jul 2021 
  

• FAFF - no action for this group yet. The use cases are at the bottom of the linked confluence page. This group is invited to review and provide input. RHL says the focus is "what do we need to do our job on the mountain". 
• Keith will be contact person for Eric/Sidney  

Jeffrey Carlin Keith Bechtol

• Status of DMTR-311

Review of any issues that have come up during the week with faro and metric computation 

Dan Taranu added photometric repeatability metrics to faro. See 

• Fix for lookup table in from Eli - 

•  Jeffrey CarlinCheck that the incorrectly specified pipelines in yaml files found by Dan have not resulted in bad results in the CMR. 06 Jul 2021 

Jeffrey Carlin Keith Bechtol

https://chronograf-demo.lsst.codes/sources/2/dashboards

Monthly reprocessing: 

• DRP metrics monthly Gen 2 (HSC RC2)
• Where is the Gen3 squash dashboard - follow up with Yusra
• Get the new test dataset running and metrics into a dashobard (or add to existing) in squash 

No results are reported in the nightly dashboards for many of the metrics

•  Simon Krughoff to look at last night's jenkins run and see why no metrics (except AM1) are showing up in squash 06 Jul 2021 

Leanne Guy

• Fall 2021 epic 

  Jira
  server JIRA serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-30748

• Backlog epic: 

  Jira
  server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-29525

Leanne Guy