• Get together, resolve bottlenecks, make progress faster, plan, design, exchange ideas

Discussion items

John joined group. Will work on Metadata / Data Access / Image Services. 


  • Two 10-SP metadata related tasks moved from AndyS to John
  • DM-1252 (4 SP) moved from AndyS to Daniel, Daniel working on code related to this ticket
  • AndyS will work on 
    • Query Metadata (new epic/stories, see DM-1487)
    • DM-621 (loader) - moving this story from Serge
  • If we want to add more scope to W15 - likely in the area of distributed loader

Distributed loader for integration tests (DM-998)

  • use existing single-node partitioner, then scp csv files produced by partitioner, and ssh <load> on each node


  • Daniel moving css to core/modules/css, includes reorganization of python modules, only czar had python so far

Metadata Store

  • reuse code from datarel, can't use datarel as-is, not generic enough
  • ideally, computation related to images should happen in afw, not in datarel or metadata store related code (apps team already agreed to deal with computation related to sources)
  • research off-the-shelf systems that might be useful (irods, what fermi did etc)
  • when importing external data sets, better to "make it butler-compatible first, then ingest"
  • metadata store needs to deal with repositories of configs
  • might need to capture in provenance which version of butler was used
  • need to discuss with SUI team if sql interface to metadata is sufficient, or web api is needed
  • different typed of images in one repo or separate? Let butler drive it: one repository per task is best
  • meaningful vs obscure file names - still tbd