Data Management
Space shortcuts
Page tree
Skip to end of metadata
Go to start of metadata

Date

Attendees

Goals

Discussion items

TimeWhoPre-meeting NotesNotes and Action Items
News, announcements and task review
CMR
Current draft of DMTR-311
  • Jeff made a ticket to use update columns and has BPS job running now that should finish soon (~30 min). The change was made in the default rather than the yaml configuration.
  • Should we be using "slot" designations in general? Depends. One does have to be careful where the slots are pointing.
  • Where does documentation exist for the columns? Not an immediate answer. We expect several of the names/columns to change.
  • The ellipticity residuals were the big change in reported metric values.
  • Need to specify high memory (40 G) when running the BPS jobs to compute metric on full tract.
  • Even with largest container in RSP, cannot load full tract catalog from object table into memory.
    • Getting the schema is a good ticket for middleware. Should be able to retrieve the schema.
    • Check to see if there is composite dataset type for the schema
  • faro should be transitioning to use object table and loading a subset of columns
  • Need to figure out how to specify the subset of columns to load in the connections class.
    • Might have to make a runQuantum function to do this, though it would be nice if could specify as argument in the input in connections class
  • Considering to use w18 instead of w14 for v22 pipelines release
    • One issue that would need to backported, but might change conda dependencies. Release management reasons.
    • There is a w18 RC2. Should we run faro on w18, Monika included. Jeff would need to re-run TE1 and TE2. Then would have w18 results in hand. 
    • w22 processing in underway
Status of reprocessing
  • w_2021_14 (RC2) 
  • w_2021_16 (DC2 gen3) (thanks Dan)
  • Now running faro with BPS, need documentation
Bugs, issues, etc of the week
  • Apparent TEx performance degradation in bands other than i DM-30359 - Getting issue details... STATUS

Development status

Review progress on the current development epic DM-26993 - Getting issue details... STATUS

Backlog epic:  DM-29525 - Getting issue details... STATUS

  • Spring 2021 epic was intended to be finished 31 May 2021
  • Next week could be an opportunity to do a comprehensive review of current epic and make priorities for the next several months
    • One high-priority item would be using object table and source table (parquet format)
AOB
  • Reference catalogs interface development: DM-30054 - Getting issue details... STATUS
  • What feedback would we like to receive from science pipeline developers from faro presentation on Tuesday 2 June?
    • What metrics do we expect pipelines to be producing in situ?
    • Visualization
    • Dedicated data structure to hold metric values in butler repo?
    • What inputs should be used for metric calculation (e.g., parquet tables vs. direct outputs of pipelines)?
    • External reference catalogs
      • Adding reference catalogs
    • External calibrations
    • Spatial scales for computing metrics (e.g., patch/tract versus other spatial partitioning scheme)
    • Communication between faro development and science pipeline development
  • Simon is updating validation_data_hsc CI dataset. Need a Gen3 dataset to use for getting started. Central 6 sensors from something like 10 visits. Calibrations and reference catalogs.
  • For science pipelines meeting,
    • Not necessarily trying to necessarily propose solutions, or get into design discussions, but try to find out who to connect with in DM to make progress
    • "What can faro do for you?"

Action items

  •