Page History

All

• Jenkin saga continues. DM-31846 Jointcal dependency. Script moved into the data repository (rc2_subset) to put the pipelines in the data repository. We don't want a situation with faro being dependent on everything because we need to run pipelines.  Woud like to get signofff from, KT. 
• Any yaml pipeline definition files run by jenkins must be outside the faro package, preferably in the data package. (Yaml) pipeline run by hand can in the faro package. 
• Keith to write down use cases for accessing metric results

Multimatch: (current code)

• Until we have different matching algorithm, we will need to use the FITS tables. We don't have an alternative to the multi match algorithm at the moment. EC is interested in working on this if it will be of use to someone. Feeling is he was looking for a driver to keep working on it.

• Not certain that the forced source photometry table will always be available. Creating coadds adn deblending on coadds are the longest steps at the moment, so if we are only looking at matched visits, will probably not want to take the time to gegnerate coadds and run forced photometry. This will be important in the commissioning context, e.g if we want to run some metrics on 10 or so visits and don't want to wait for the forced source tables. 

• Three issues here: 
  1. Timing of the availability of forced photometry
  2. Scientific issue of whether we want forced photometry versus independent detection and centering
  3. Multimatch algorithm - we are planning to move away from this anyway to an n-way matching algorithm that does not assume an afw_table input. 

CS: Proposes to get off afw tables as soon as posible.  All the sky processing doess have coadds. We can avoid the isssue of the forced source Implement using the forcedSourceTables.  We can support the commissioning use case later. 

KB: Can we spend a few hours to remove afw_tables from multi-match. For bright isolated stars, we can use multi-match.  Aggregate function is very slow, can take minutes to compute the average RA. 

KSK: search for multimatch in lsst org - 15 hits - -most are faro, vailidate_drp, a few in notebooks. 

•  Leanne Guy , Keith Bechtol  will follow up to remove afw_tables from multi-match.  30 Nov 2021 

External calibrations: 

• Parquet Source files do not have these external calibs applies
• Need to do this if we want to apply external calibrations to source tables. Will allow us to bypass the stage of reading in the external calibrations 
• These are input to FGCM - needs to run as an afterburner for now. 
• Not high priority for now but we should do this. Ticket on backlog

  Jira
  server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-31855

Jeffrey Carlin Keith Bechtol

• Nightly CI with new rc2_subset
• w_2021_34 RC2
• w_2021_36 DC2
  • From DRP QA meeting - bad astrometry and photometry metrics for gen2to3 as compared with native gen3 processing – see 

    Jira
    server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-31762

  • Probably external calibrations, afterburner, check that faro is picking up the correct inputs. 
• w_2021_38 RC2
  • In progress. This will be the first reprocessing to include `forcedSourceTable` parquet files.

  • Jira
    server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-31795

Colin Slater 

Leanne Guy  

Faro for DP0.2

•  Hsin-Fang reports that the high level workflow will be frozen on 29 Sep 2021 . We need to also have the faro pipelines that will be run defined and part of the workflow. Practically it’s adding faro pipelines into those steps in DRP.yaml  so that faro runs integrated with DRP.

• rtn-001.lsst.io, 2.1.4 High level workflow of workflows
• W_2021_40 was planned to be the starting point for DP0.2, including the cleaned up parquet schema that was presented at the DM-SST meeting. See RFC-807. 
• There will be single tract test V&V run using weekly 40.  
• Add clustering in mid October - also any fixes from V&V run - then make candidate for the step1 processing pilot run  (w_42)

Leanne Guy

• Fall 2021 epic

  Jira
  server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-30748

• Backlog epic: 

  Jira
  server JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-29525

• Jeff to focus on matched source conversion to parquet tables
  • Switch to using forcedSourceTable instead of doing matching of single-epoch detections?
• Using Object tables 

  Jira
  server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-31459

• Astrometry –  wait for Object tables in parquet

  Jira
  server JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-31673

• faro documentation:

  Jira
  server JIRA serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-25839

  
  

Leanne Guy