Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

ItemWhoPre-meeting notesNotes and  Action Items 
News, announcements and task review

DM Development Workflow.

Slides on faro package presented at DM Science Pipelines meeting last week

  • Science Pipelines would like faro metrics to be routinely shipped to SQuaSH and appearing on chronograf dashboard
  • Question of best way to handle situations when the metric definition (algorithm for calculating) changes. Annotate in SQuaSH dashboard.

Development workflow: 

  • Reviewer should not approve until successful
  • Make sure to rebase with respect to matter
  •  Start policy to ensure successful Jenkins run prior to merge of faro PRs.   
  • Need to routinely ship faro metrics to SQuaSH and display on chronograf dashboard
    • Proposed approach to teach Yusra to run
    • How do we make this automated? Need a script to automate. We can help Yusra to provide information about what needs to go into the script.
    • For the metrics of interest, the definitions exist.
  •  Simon Krughoff to schedule a time to meet with Yusra or designate from DRP team to see how to to ship faro metrics to squash.   
CMRCurrent draft of DMTR-311
  • Jeff spoke to Yusra and they decided to use w14 for v22.0.0 and backport the scarlet fix. Yusra wants to see a direct comparison of faro metrics on direct Gen3 processing and Gen2 to Gen 3 conversion version.
    • However, faro won't run on the converted repo with changes made for external calibration support. Eli is working on a fix.
    • Yusra is interested to confirm that Gen2 to Gen3 is equal to or slightly better than the native Gen3 version (Gen3 version does not yet have all the features of Gen2 to Gen3 conversion version)
Bugs, issues, etc of the weekAll 

Review of any issues that have come up during the week with faro and metric computation 

 : do we do this in faro?

What is the status in faro of these old validate_drp tickets: 

Jira
serverJIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId9da94fb6-5771-303d-a785-1b6c5ab0f2d2
keyDM-16555
: this error appears to have been propagated to faro.

Jira
serverJIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId9da94fb6-5771-303d-a785-1b6c5ab0f2d2
keyDM-13758

  • Jira
    serverJIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId9da94fb6-5771-303d-a785-1b6c5ab0f2d2
    keyDM-30656
     to fix DM-16555  in faro . Can we get this fixed before issuing the CMR
  •  Jeffrey Carlin to investigate and decide if relevant for faro  
  • Jira
    serverJIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId9da94fb6-5771-303d-a785-1b6c5ab0f2d2
    keyDM-13758
     - do the filtering before assembling the matched catalog 
Status of reprocessing 
  • What versions are we reprocessing now? 
  • What will be the weekly version for R22.0.0?
  • Complete CMR for R22.0.0 – see notes above for w14
  • w18. One of the tracts failed in the coadd step. Monika offered to re-run. 2 of 3 tracts were fine. Third made single-frame but not coaddition step.
  • w22 processing is already running. Note that since Eric Morganson left the project, Morgan Schmitz is trying to get up to speed with RC2 processing.
  • Simon is getting close to have small 6 chips in 9 visits for 5 bands. Successfully processed all through coadd except g band. Picking better visits should fix. All using butler import/export tools. Native Gen3. This would replace validate_drp_hsc. Size is 14 GB including reference catalogs, calibrations. Repo contains raws. Simon has done the fix to get reference catalogs from only the relevant portion of the sky.
Development status 
  • Spring 2021 epic was intended to be finished 31 May 2021
  • Next week could be an opportunity to do a comprehensive review of current epic and make priorities for the next several months
    • One high-priority item would be using object table and source table (parquet format)

Review progress on the current development epic

Jira
serverJIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId9da94fb6-5771-303d-a785-1b6c5ab0f2d2
keyDM-26993

Backlog epic: 

Jira
serverJIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId9da94fb6-5771-303d-a785-1b6c5ab0f2d2
keyDM-29525

  •  Keith Bechtol Check to see whether we can access the schema of tract object tables (who?)  
AOBNext meeting 2021-06-15.

...