Date and Location

, 11am-12pm pacific.

 ls.st/sup

Attendees

Goals

  • To start getting on the same page with regards to butler, with focus on what butler really is, what it does, what its roles are, what APIs is has, and most importantly what it will and will not do in the future.

Discussion items

It'd be great if each team could cover features you think you "must have" in butler that are not available today, and features that would be nice-to-have but are not critical.

TimeItemWhoNotes
5minIntroK-T or NateP or Jacek

 

10minUW 
  • Easily specify new outputs from tasks (no need to edit obs_ packages)
  • butler.get should fail immediately if the data is not available
  • Easily iterate over partial IDs for all dataset types
  • Simplify the way IDs are normalized (?), and document the system better; also preferably make it easier to know when you have an ID that has been normalized or not.
  • Handling of bitemporal calibration products including camera descriptions. N.b. calibration products can be lots of different things: objects, images, telemetry data, sky model, etc.
  • Butlerized access to logs, configs, stack version, etc.
10minPrinceton DRP Butler Requirements / Feature Requests
10minIPAC SUI / Level 3 and SuperTask feedback on the Butler
10minSQUARE 
10minNCSA

 

 
 Architecture 

Tim Jenness : I feel strongly that we should have two persistable file format options designed in to the butler (FITS and HDF5 say) so that we can be sure we have a flexible design that is not wedded to a single file format.

Unknown User (npease): 

  • DM-4543 - Getting issue details... STATUS
  • DM-4542 - Getting issue details... STATUS
  • DM-4541 - Getting issue details... STATUS (not butler work, has an AFW component tag)
 Discussion 
  • Carry technical discussions via community
  • Next meeting: Tuesday Dec 8, 12:30pm, via bluejeans. Focus: (a) check if created stories adequately cover all requests, (b) finalize prioritization

DRP Butler Requirements / Feature Requests

SUI / Level 3 and SuperTask feedback on the Butler

Random notes

Butler (not-done) stories:

T Key Summary Assignee Reporter P Status Resolution Created Updated Due
Refresh

Action items

  • Unknown User (npease), turn the requirements discussed / input received into JIRA stories
  • Identify core architectural issues
  • Design core architectural issues
  • Write a comprehensible high level design document
  • Create interface definition (we will defer remote butler & URL interface definition)
    • Application Interface
    • Configuration Interface
    • Back-end interface. Include what can be plugged in, and where.
      • transport
      • serialization
      • storage
  • Unknown User (npease) and Kian-Tat Lim - propose prioritization of all butler stories
  • Work on top-priority item for each group:
    • Princeton: DM-4181 - Getting issue details... STATUS (epic S16) (Serge Monkewitz is working on kind-of-related DM-3472 - Getting issue details... STATUS )
    • SUI: Multiple repositories: finish defining, create story, work on partial implementation TBD.
      • DM-4625 - Getting issue details... STATUS
      • Implement  DM-4682 - Getting issue details... STATUS (epic W16) (might get broken into 2 stories and delivered over 2 sprints)
    • UW:  DM-4168 - Getting issue details... STATUS  (epic Butler W16), also "if time":  DM-4180 - Getting issue details... STATUS  (epic S16)
    • Tuscon: define interface for  DM-4555 - Getting issue details... STATUS (epic S16, assigned to BVK)
    • NCSA: (nothing at this time)

1 Comment

  1. Gregory Dubois-Felsmann

    An additional provocation about the Butler's ability to understand associations ("Capability 3" on the SUI inputs child page):

    Does this connect in any way to provenance?  Should it?  Consider:

    • Retrieve all calibrated images associated with (overlapping) patch P; versus
    • Retrieve all calibrated images actually used to generate patch P of a released coadd (i.e., not including ones that were rejected for some reason).

    or...

    • Get the synthetic flat that should be used for visit X; versus
    • Get the synthetic flat that was actually used to generate the calibrated image Y from that visit.