Jenkin saga continues. DM-31846 Jointcal dependency. Script moved into the data repository (rc2_subset) to put the pipelines in the data repository. We don't want a situation with faro being dependent on everything because we need to run pipelines. Woud like to get signofff from, KT.
Any yaml pipeline definition files run by jenkins must be outside the faro package, preferably in the data package. (Yaml) pipeline run by hand can in the faro package.
Keith to write down use cases for accessing metric results
Bugs, issues, topics, etc of the week
All
It looks like `multiMatch` assumes all inputs are afw_tables, which makes it difficult (or inefficient) to use it with the `sourceTable` (parquet) inputs (see this Slack thread). Are the `forcedSourceTable`s sufficient for our needs?
External calibrations have not already been applied to fluxes from the `sourceTable` parquet tables. You still need to read in the FGCM and jointcal calibrations and apply them. Furthermore, before applying the external calibs, you must "undo" the `LocalPhotoCalib` correction that has already been applied to the fluxes in `sourceTable`s (Slack thread here).
Yusra suggested that "somebody" should write an afterburner that applies the external calibs and inserts calibrated fluxes/positions into the tables. Could one of us be that "somebody"?
Until we have different matching algorithm, we will need to use the FITS tables. We don't have an alternative to the multi match algorithm at the moment. EC is interested in working on this if it will be of use to someone. Feeling is he was looking for a driver to keep working on it.
Not certain that the forced source photometry table will always be available. Creating coadds adn deblending on coadds are the longest steps at the moment, so if we are only looking at matched visits, will probably not want to take the time to gegnerate coadds and run forced photometry. This will be important in the commissioning context, e.g if we want to run some metrics on 10 or so visits and don't want to wait for the forced source tables.
Three issues here:
Timing of the availability of forced photometry
Scientific issue of whether we want forced photometry versus independent detection and centering
Multimatch algorithm - we are planning to move away from this anyway to an n-way matching algorithm that does not assume an afw_table input.
CS: Proposes to get off afw tables as soon as posible. All the sky processing doess have coadds. We can avoid the isssue of the forced source Implement using the forcedSourceTables. We can support the commissioning use case later.
KB: Can we spend a few hours to remove afw_tables from multi-match. For bright isolated stars, we can use multi-match. Aggregate function is very slow, can take minutes to compute the average RA.
KSK: search for multimatch in lsst org - 15 hits - -most are faro, vailidate_drp, a few in notebooks.
From DRP QA meeting - bad astrometry and photometry metrics for gen2to3 as compared with native gen3 processing – see
DM-31762
-
Getting issue details...STATUS
Probably external calibrations, afterburner, check that faro is picking up the correct inputs.
w_2021_38 RC2
In progress. This will be the first reprocessing to include `forcedSourceTable` parquet files.
Hsin-Fang reports that the high level workflow will be frozen on . We need to also have the faro pipelines that will be run defined and part of the workflow. Practically it’s adding faro pipelines into those steps in DRP.yaml so that faro runs integrated with DRP.
W_2021_40 was planned to be the starting point for DP0.2, including the cleaned up parquet schema that was presented at the DM-SST meeting. See RFC-807.
There will be single tract test V&V run using weekly 40.
Add clustering in mid October - also any fixes from V&V run - then make candidate for the step1 processing pilot run (w_42)
Jeff has tested this in DRP.yaml in rc2_subset. Separated into SFP and coadd processing and ran 2 groups at the end.
i.e compute SVP metrics at the stage of SVP rather than waiting for everything to finish
Wants to do more testing in CI imsim DRP.yaml before going with it.
Keith Bechtol to make a ticket to better understand mapping of these camera and calibration products characterization efforts to verification documents and the focus of these efforts. Discuss with the SCLT