Data Management
Space shortcuts
Page tree

Versions Compared

Old Version 8

changes.mady.by.user Leanne Guy

Saved on

compared with

New Version Current

changes.mady.by.user Leanne Guy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

ItemWhoPre-meeting notesMinutes
News, announcements and task review

All

Jeff is on leave for 2 weeks. 

Welcome to Michael Martinez, grad student working with Keith at UW-Madison

  • Task review
    • Chris to present next week
    • Still an open question about the precise definition of the stellar locus. Leave that task open for now.
    • There are high-level discussions about the organization of SIT-Com. Expect more clarity in a few weeks. We will return to several of the open tasks.
    • Consolidated database. Different from reformatted
      • Keith to find ticket
    • Verification planning for commissioning
      • Keith is currently working a branch for SITCOMTN-006
    • Put in place a process to understand relationship between camera verification and general science verification

CI status, bugs, issues of the week

All
  • Recent changes to the order of matching and filtering ad possibly other things (scarlet)  have resulted in changes to metric values. We need to investigate 
  •  Leanne Guy  to create a ticket (or check if we have one)  
Faro Development status 

Leanne Guy

  • Spring 2022 epic 
    Jira
    serverJIRA
    columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId9da94fb6-5771-303d-a785-1b6c5ab0f2d2
    keyDM-33385
  • Backlog epic: 
    Jira
    serverJIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId9da94fb6-5771-303d-a785-1b6c5ab0f2d2
    keyDM-29525
  • Progress on conversion to parquet
  • Ability to dump the in-memory python objects to a shelve is now merged
    • Also ideal location to have input datasets for unit tests. 
    • Only run in debug/analysis mode  - not operational context
  • Open PR for wPerp conversion to Parquet table inputs 
    • Will run as part of step 5 in DP0.2 - want to get it backported. 
    • IN2P3 cannot run old wPerp. 
  • Matching algorithms and matched isolated bright stars dataset type - Eli developed a matcher for FGCM 
    • Has criteria to select isolated stars
    • Outputs two catalogs:
      • Set of primary sources
      • Measurements for primary sources (all collated and associated with band, easy to parse)
    • Fast algorithm  - could be used for our repeatability metrics on bright stars 
    • No prior on the delta mag - could deal with stellar matching well but don't expect it to work for galaxies
    • Mainly interested in stellar matching for the purpose of single-epoch repeatability measurements.
    • Potentially tweak the filters.
    • Try it out on the upcoming 1deg AuxTel survey data 
    • Currently implemented as pipe_task. Peter has run it separately. Proposal is to use those tables.
  •  Leanne Guy Create ticket to compare outputs from current matcher and FGCM matcher. Assign to Peter and Keith  

Data processing campaigns  

Jeffrey Carlin Keith Bechtol

  • AT data reduction
    • new collection with gri on a reasonably dense field
    • ED trying to understand tables & outputs  before running visit & detector level data through faro.
DP0.2

Colin Slater 

Leanne Guy  

  • DP0.2 currently running into an issue where matchedCatalogTask runs for >2 hours without emitting any log messages. The Panda monitoring system assumes that means it has crashed or is in an infinite loop or something, so it terminates the job.
  • Had to switch to running the non-faro parts of step3
  • We now have a reprieve from that limit (as of an hour ago), so we might be able to resume running it as normal?
    • CTS: Confirmed, it's working fine now.
  • Hypothesis is that the butler.get() at the start of runQuantum is taking all the time.
  • Deferred dataset loading might also be a good idea in general, and would avoid a long block of time spent in single butler.get()
    • Middleware implemented a fix to emit logs messages during long butler.get() calls.

Ran out of time  last week - pick up today?

  • CTS: I think this is now superseded. Faro is running as-is in step3, and the investigations into Eli's matched star catalog might replace a lot of this.
  • Collection id for g',r',i' products generated from single-frame processing: 

    'u/edennihy/tickets/CAP-851_20220307a'

  • A quick example query to isolate the survey data using the program key:
    • program = "'LATISS_E6A'" 
    • cal_exps = sorted(registry.queryDatasets('calexp', collections=collection, instrument='LATISS',where="exposure.science_program={}".format(program)))
  • Process - who do we need work with to understand issues - zeropoint, meaning of columns
    • e.g what is our 10sigma depth? How to understand the flux values and convert to magnitude  
      • CTS: look to how to do a full processing  - investigate at co-work 
  •  
AOB

Leanne Guy

Potential co-working session ideas here

Next meeting   

...