Embedded DataIds

With: Jim Bosh, Nate Pease

Get more details from Jim about per-CCD PSFs that are used as components of a coadd-level PSF, the coadd dataset stores the data IDs of all of its inputs.

Non-Posix Storage Backends (aka 'Storage and Format Refactor)

With: Fabio Hernandez, Nate Pease, Maria Patterson, Simon Krughoff

Discuss mechanisms currently provided by the Butler for using non-POSIX storage backends. (Currently there are none, but we can review the current thinking described in Butler Storage and Format Refactor and discuss requirements)

Maria Patterson has a little bit of experience from a previous project operating S3-compatible storage at the near petabyte scale (specifically, Ceph's radosgw) as a data repository and would be glad to share knowledge/experience.

There is a person at UW trying to run the command line tasks on Spark and she ended up un-butling the tasks. Simon is interested in knowing whether a backend suitable for Spark is even a reasonable thing to ask for.

Overlap Between Storage & Format Refactor and Composites

With: Jim Bosh, Nate Pease

Discuss how these 2 features overlap in terms of use cases and how their proposed implementations affect each other.

Butler Storage and Format Refactor

Requirements and Design for Composite Datasets in Butler

Intermediates in assembler 

In Jim's comments on the page Requirements and Design for Composite Datasets in Butler:
I am a bit worried that when we redesign the low-level serialization layer, we'll want to change some of this. For instance, if we decide we need an intermediate format (e.g. nested tuples+dicts+numpy.ndarrays) to mediate between the kinds-of-objects and kinds-of-storage dimensions, I could imagine the assembler interface here using that intermediate format instead of having direct access to a butler. Getting component datasets working is a much higher priority right now than refactoring the low-level serialization layer, but we should keep this in mind.

I think this is addressed by Custom Assemblers which I had neglected to write about before this comment was made. Need to touch base though.

Use Cases for Requirements

With: Jim Bosh, John Swinbank, Nate Pease

Discuss how to capture an appropriate amount of high-level use case when writing Butler requirements.

Maybe make an initial pass at writing down those use cases for "Storage & Format" and "Composites".