Data Management
Space shortcuts
Page tree
Skip to end of metadata
Go to start of metadata

Date

 

Location

Zoom alias : https://ls.st/rsv

Zoom:  https://noirlab-edu.zoom.us/j/96120781994?pwd=bCtpT3Q3b2RsU1ZBRUFaSnUrZXo3Zz09#success

Attendees


Regrets

Metric tracking dashboards

Rubin science pipelines weekly change log and  Github repo: https://github.com/lsst-dm/lsst_git_changelog

Status of jenkins jobs and slack channle for notifications:  #dmj-s_verify_drp_metrics slack channel 

Discussion items

ItemWhoPre-meeting notesMinutes
News, announcements and task review

All

Jeff is on leave for 2 weeks. 

Welcome to Michael Martinez, grad student working with Keith at UW-Madison

  • Task review
    • Chris to present next week
    • Still an open question about the precise definition of the stellar locus. Leave that task open for now.
    • There are high-level discussions about the organization of SIT-Com. Expect more clarity in a few weeks. We will return to several of the open tasks.
    • Consolidated database. Different from reformatted
      • Keith to find ticket
    • Verification planning for commissioning
      • Keith is currently working a branch for SITCOMTN-006
    • Put in place a process to understand relationship between camera verification and general science verification

CI status, bugs, issues of the week

All
  • Recent changes to the order of matching and filtering ad possibly other things (scarlet)  have resulted in changes to metric values. We need to investigate 
  • Leanne Guy  to create a ticket (or check if we have one)  
Faro Development status 

Leanne Guy

  • Spring 2022 epic  DM-33385 - Getting issue details... STATUS
  • Backlog epic:  DM-29525 - Getting issue details... STATUS
  • Progress on conversion to parquet
  • Ability to dump the in-memory python objects to a shelve is now merged
    • Also ideal location to have input datasets for unit tests. 
    • Only run in debug/analysis mode  - not operational context
  • Open PR for wPerp conversion to Parquet table inputs 
    • Will run as part of step 5 in DP0.2 - want to get it backported. 
    • IN2P3 cannot run old wPerp. 
  • Matching algorithms and matched isolated bright stars dataset type - Eli developed a matcher for FGCM 
    • Has criteria to select isolated stars
    • Outputs two catalogs:
      • Set of primary sources
      • Measurements for primary sources (all collated and associated with band, easy to parse)
    • Fast algorithm  - could be used for our repeatability metrics on bright stars 
    • No prior on the delta mag - could deal with stellar matching well but don't expect it to work for galaxies
    • Mainly interested in stellar matching for the purpose of single-epoch repeatability measurements.
    • Potentially tweak the filters.
    • Try it out on the upcoming 1deg AuxTel survey data 
    • Currently implemented as pipe_task. Peter has run it separately. Proposal is to use those tables.
  • Leanne Guy Create ticket to compare outputs from current matcher and FGCM matcher. Assign to Peter and Keith  

Data processing campaigns  

Jeffrey Carlin Keith Bechtol

  • RC2/DC2 reprocessing epic : DM-26911 - Getting issue details... STATUS
    • w_2021_40 RC2:   DM-32248 - Getting issue details... STATUS
    • w_2021_44 DC2 : DM-32391 - Getting issue details... STATUS
  • AuxTel - Erik to give update
  • AP/DiffIm
  • AT data reduction
    • new collection with gri on a reasonably dense field
    • ED trying to understand tables & outputs  before running visit & detector level data through faro.
DP0.2

Colin Slater 

Leanne Guy  

  • DP0.2 currently running into an issue where matchedCatalogTask runs for >2 hours without emitting any log messages. The Panda monitoring system assumes that means it has crashed or is in an infinite loop or something, so it terminates the job.
  • Had to switch to running the non-faro parts of step3
  • We now have a reprieve from that limit (as of an hour ago), so we might be able to resume running it as normal?
    • CTS: Confirmed, it's working fine now.
  • Hypothesis is that the butler.get() at the start of runQuantum is taking all the time.
  • Deferred dataset loading might also be a good idea in general, and would avoid a long block of time spent in single butler.get()
    • Middleware implemented a fix to emit logs messages during long butler.get() calls.

Ran out of time  last week - pick up today?

  • CTS: I think this is now superseded. Faro is running as-is in step3, and the investigations into Eli's matched star catalog might replace a lot of this.
  • Collection id for g',r',i' products generated from single-frame processing: 

    'u/edennihy/tickets/CAP-851_20220307a'

  • A quick example query to isolate the survey data using the program key:
    • program = "'LATISS_E6A'" 
    • cal_exps = sorted(registry.queryDatasets('calexp', collections=collection, instrument='LATISS',where="exposure.science_program={}".format(program)))
  • Process - who do we need work with to understand issues - zeropoint, meaning of columns
    • e.g what is our 10sigma depth? How to understand the flux values and convert to magnitude  
      • CTS: look to how to do a full processing  - investigate at co-work 
  •  
AOB

Leanne Guy

Potential co-working session ideas here

Next meeting   


List of tasks (Confluence)

DescriptionDue dateAssigneeTask appears on
  • Leanne Guy to talk to Science Pipelines (Yusra) about when do this transfer  
19 Oct 2021Leanne Guy2021-09-28 Science Metric Development Agenda and Meeting notes
  • Leanne Guy arrange to disucss at a future meeting if there are metrics from PDR3 & this paper that we might want to include in faro.   
26 Oct 2021Leanne Guy2021-08-31 Science Metric Development Agenda and Meeting notes
  • Colin to ask about capturing ideas for improvement to the stellar locus algorithm   
30 Nov 2021 2021-11-09 Science Metric Development Agenda and Meeting notes
  • Colin Slater to make a preliminary draft agenda for a workshop to clarify visualization use cases for science verification and validation
 Colin Slater2022-04-19 Science Metric Development Agenda and Meeting notes
  • Jeffrey Carlin to review metric specification package organization and the relationship to formal requirements documents
 Jeffrey Carlin2022-04-19 Science Metric Development Agenda and Meeting notes
  • Keith Bechtol Schedule a time to have focused discussion on verification package, potentially next status meeting
 Keith Bechtol2021-09-14 Science Metric Development Agenda and Meeting notes
  • Keith Bechtol to make a ticket to better understand mapping of these camera and calibration products characterization efforts to verification documents and the focus of these efforts. Discuss with the SCLT
 Keith Bechtol2021-09-14 Science Metric Development Agenda and Meeting notes