All
- Task review
- SPIE faro paper
- Open topics from last week
- Running faro on calibs / ISR verification
- Metrics for near-realtime data quality analysis to support AuxTel imaging campaigns (we could ask at FAFF meeting later today)
- Erik making progress on TEx metric
- Documentation
- PR (tickets/DM-25839)
- For the TEx metric utility reimplementation, it is functionally working for TE3 and TE4, and now trying to understand numerical differences between the parquet table implementation and afw table implementation.
- Documentation
- rc2_subset is not suitable to use for CI testing. Not currently configured with tests, and so building the package will not be a useful CI test for faro. Current EUPS packages to run during Jenkins testing is still "lsst_distrib lsst_ci ci_hsc_gen3 ci_imsim"
- Request that we clean up "test" annotations, Simon will delete.
Bugs, issues, topics, etc of the week
From Jeff on Thursday last week: Just a note to let RSV folks know that verify_drp_metrics has failed for two days in a row, and will continue to do so until Simon and I can get to it on Mon./Tues.. It’s related to a change we made that added DIA processing into the pipeline (and is thus failing due to the issue described in DM-31777), and we’re considering solutions (but limited on time, and I’ll be out of office tomorrow).
- Current status?
Data processing campaigns
- AuxTel
- AP DiffIm
- Currently failing. Jeff is working on it. Seems like failing in faro step to make the matched catalogs. Not finding FGCM photocalibs. Not easy to test in part because it takes a few hours to get to the point where the failure occurs. Save the sqllite registry to another name. Then you can repeatedly run the steps afterwards.
- Ask Jim dm-middleware-support the recommended way to do this
- Merlin finished Gen2 config changes. Might be a one-liner. Propose to wait until Merlin does the last bit. All the thinking is done. The reduce-auxTel ticket is DM-32141.
Reprocessing status and metrics review
- Nightly CI with new rc2_subset
- RC2/DC2 reprocessing epic :
- w_2021_38 RC2:
- w_2021_36 DC2 :
- From monthly DRP metrics. Need to push on getting all the metric calculations working with parquet tables. Metrics are not being calculated because they require too much memory. We should push
Faro for DP0.2
- Colin mentioned yesterday that doing some test processing runs on the IDF. Included in pipelines that are being run are the faro tasks. We should soon have metrics to start monitoring. We should set up a dashboard to monitor.
- Fall 2021 epic
- Backlog epic:
- Peter has started working on AA1 metric implementation (DM-29368)
- Open PR for the force source implementation (DM-31460)
- Open topic for another day: metrics that require telemetry (EFD) or correlation with instrument coordinates
- Might be more appropriate as an afterburner using database query than run within faro?
- We might consider bending the rules to allow database queries within tasks
- We could imagine building binned summary statistic in faro by tract, etc.
Potential co-working session ideas here
