If going to have a special session with co-work, we should advertise. It is no longer co-work if we have a one-hour presentation. Perhaps those topics are more appropriate for the Tuesday status meeting. Sentiment that this should be unstructured time. Don't expect deep dives into technical topics to be very regular.
Bugs, issues, topics, etc of the week
All
We have metrics in chronograph!!!! Horray.
But we have issues with the dashboard.
Metrics and datasets collided and end up on wrong plots. There is something amiss about the selections that needs to be sorted out.
Probably not filtering correctly on the dataset tag.
Select 7d + tract 9813 to get only faro + rc2_subset metrics.
Can we add the spec. thresholds to the chronograph plots? Not easy with this version. Moving to chronograph 2.0 on a ~ 6month timescale.
Could potentially add data for specs, but unfortunately chronograph would not treat lines as being special in any way other than mousing over to see label. Maybe we should think about report tooling that queries from influxDB. There effort into visualization in influx2.
We will have to do a bulk shift to a new table not named validate_drp. All science pipelines folks are using that package as well. Another option is generating a new table that is a copy and starting from scratch populating with metric values. There are ways a renaming a table. SELECT * into (new table name). Then everyone would have to put new results in new table.
Leanne Guy to talk to Science Pipelines (Yusra) about when do this transfer
Jeff has tested this in DRP.yaml in rc2_subset. Separated into SFP and coadd processing and ran 2 groups at the end.
i.e compute SVP metrics at the stage of SVP rather than waiting for everything to finish
Wants to do more testing in CI imsim DRP.yaml before going with it.
Planning to put ticket into review and get it in today or tomorrow.
DM-31848
-
Getting issue details...STATUS
Colin wants to better understand the errors seen running on full end-to-end processing on <dataset>, can we test on other datasets, or separate out the steps to look at each.
Can we run the faro steps on the weekly 36 processed data? Run faro based on the products of DRP.yaml would be a useful exercise.
Parquet conversion
tract and patches:
DM-31459
-
Getting issue details...STATUS
Keith has picked up some of Erik's work on converting to using parquet tables. Keith took the branch and added capability to selectively load columns from the specific band. Need this to compute a per-band metric.
Difficult to run TE3/4 or stellar locus based on the parquet files in the next 48hrs. Fallback is that we continue to use the FITS files.
Column renaming RFC is yet to land - are we depending on anything that will change, e.g., psFlux → psfFlux . Most changes are in the Object table.
Decision: FITS files will be our basis for DP0.2 process. Continue to push ahead and complete the migration to parquet.
Keith Bechtol to make a ticket to better understand mapping of these camera and calibration products characterization efforts to verification documents and the focus of these efforts. Discuss with the SCLT