This page has been completely rewritten from its original form, and the terminology has changed. In particular, because we've settled on an approach in which all datasets always have an entry in the Registry's Dataset table, we have repurposed "virtual" to mean something different.
Simplified Exposure as an Example
I'll use the Exposure StorageClass for most examples. For the purposes of this page, we'll assume it has the following simplified definition (pseudocode; not actual APIs):
Wcs = StorageClass(...)
Image = StorageClass(...)
MaskedImage = StorageClass(
components={
"image": Image,
"variance": Image
},
...
)
Exposure = StorageClass(
components={
"maskedImage": MaskedImage,
"wcs": Wcs,
"image": "maskedImage.image", # aliases to more deeply-nested components
"variance": "maskedImage.variance"
},
...
)
Terminology
composite: a Dataset or DatasetType whose StorageClass defines a set of discrete named child datasets, called components
parent: synonym for composite
component: a Dataset or DatasetType that may be accessed as a child of a composite (in some cases may also be accessed in other ways)
child: synonym for component
virtual: a Dataset or DatasetType that is defined by its relationship to one or more other Datasets/DatasetTypes.
concrete: not virtual
immediate: all content is written in a single call to Butler.put
deferred: content is written via multiple calls to Butler.put (possibly in different SuperTasks) and associated later (possibly in yet another SuperTask) via a call to Butler.link.
This leads to four fundamental kinds of Dataset[Type]s:
- concrete (always immediate, may be a component or a composite or neither): this is a regular Dataset that is itself written by a single
Datastore.putcall and read by a singleDatastore.getcall. Writing a concrete composite also creates virtual components. Examples: a WCS written on its own, an Exposure written all at once into a single file1, or a Wcs written to its own file when writing an Exposure to multiple files. - virtual component (always immediate): a Dataset defined automatically when its parent Dataset is written. Example: the Wcs of Exposure written at once into a single file1.
- immediate virtual composite: a Dataset written at once by recursively writing its components. Example: an Exposure written by writing all of its components into separate files (which could include writing an immediate virtual composite MaskedImage that in turn writes concrete Image and Variance).
- deferred virtual composite: a Dataset defined by associating pre-existing Datasets that are interpreted as its components. Example: an Exposure written by combining a MaskedImage from a different Exposure with a Wcs that was written on its own.
1. I'm saying "file" rather than "Dataset" here (and below) both to provide a clearer example and because I think concrete composite vs. immediate virtual composites is how we'll want to implement single-file Exposure vs. multi-file Exposure. This design doesn't actually guarantee that a Datastore will write a concrete composite as a single file (after all, it could even write some/all of it to a SQL database instead), but Datastores that actually do write files shouldn't need to be able to split up concrete composites into multiple files themselves. The design does guarantee that an immediate virtual composite will not be written as a single file.
Principles
- All Datasets have an entry in the Registry's Dataset table. This implies that a composite Dataset is more than the sum of its components: it also includes (Registry) information to associate them.
- Any provenance graph (both what's recorded after processing and the QuantumGraph produced by Preflight) must contain nodes for both composite and components, because:
- a SuperTask may consume only some components of a composite, so all component nodes must be in the graph;
- a deferred virtual composite must be created explicitly, so it cannot be considered implicit in the graph;
- whether a particular composite DatasetType is defined as concrete, virtual, immediate, or deferred should not affect what procesing is done, so we cannot include just some composite nodes in the graph.
- All information needed to read a Dataset is saved at the level of the Dataset (in some combination of Registry or Datastore). No information necessary for reading a Dataset is stored at a Datastore-wide or Registry-wide level, and it should never be necessary to configure a Butler a certain way in order to read something.
- It must be possible to change whether a particular DatasetType is written as concrete, virtual, immediate, or deferred by changing only the Butler/Datastore configuration provided when initializing a client.
- As a result, any composite StorageClass must be writeable as concrete (with virtual components), immediate virtual, or deferred virtual; the StorageClass itself shall not be specialized to one of these choices.
- No persistent Registry or Datastore state may be changed when controlling how a composite DatasetType is written.
Permitted Combinations
- A virtual component must be a part of exactly one concrete composite. For example, if Exposure A and Exposure B are each written as a single file, then A.wcs cannot also be a component of B.
- A virtual component may be a part of one or more deferred virtual composites. For example, if Exposure A and Exposure B are each written as a single file, then an Exposure C may be defined such that C.wcs = A.wcs and C.maskedImage = B.maskedImage.
- A virtual component may be a part of at most one immediate virtual composite if and only if its concrete parent composite is also a component of that immediate virtual composite. For example, if Exposure D is a virtual immediate composite, and its maskedImage is concrete, writing D writes D.maskedImage to a single file (probably1), and then associates the "image" virtual component D.maskedImage.image with D as D.image.
- A concrete composite always contains virtual components. For example, writing an Exposure A as a single file always implies that a A.wcs is a valid dataset (though it may be permitted to be None/null).
- A concrete composite may not contain concrete components.
- A concrete composite may not contain virtual composites.
- An immediate virtual composite may contain one or more concrete components. For example, if Exposure D is an immediate virtual composite, its maskedImage and wcs components will be written (when D is put) as separate files.
- An immediate virtual composite may contain one or more virtual components as long as it also contains their concrete composites (this is a restatmenet of (3)).
- An immediate virtual composite may contain other immediate virtual composites. For example, if Exposure E is an immediate virtual composite, its maskedImage component may also be an immediate virtual composite, which means that the image and variance will each be written as a distinct concrete datasets (i.e. separate files) when E is put.
- An immediate virtual composite may not contain deferred virtual composites.
- A deferred virtual composite may contain one or more concrete components. For example, we could write a stand-alone Wcs F, then later define an Exposure G such that G.wcs is F.
- A deferred virtual composite may contain one or more virtual components. Those virtual components must still have concrete composite parents, but those concrete composite parents need not be children of the deferred virtual composite. This is a restatement of (2).
- A deferred virtual composite may contain one or more immediate virtual composites. For example, we could write a MaskedImage H as an intermediate virtual composite, resulting in F.image and F.variance being written as separate files. We could then define an Exposure J such that J.maskedImage is H.
- A deferred virtual composite may contain one or more other deferred virtual composites. For example, we could write two Images K and L, then define a MaskedImage M such that M.image=K and M.variance=L, and then define an Exposure N such that N.maskedImage=M (and N.image=K and N.variance=L).
Configuration
Registry
When a DatasetType with a composite StorageClass is declared to a Registry, DatasetTypes for each of the named components are also declared, with names constructed as "{parent-dataset-type-name}.{component-name}", the same DataUnits types as the parent, and StorageClasses that are the same as what would be used to write the component as a standalone Dataset.
For example (pseudocode):
# Given the StorageClasses defined above, this line:
registry.registerDatasetType(CalExp=DatasetType(StorageClass=Exposure, DataUnits=("Visit", "Sensor")))
# implies:
# registry.registerDatasetType(CalExp.wcs=DatasetType(StorageClass=Wcs, DataUnits=("Visit", "Sensor")))
# registry.registerDatasetType(CalExp.maskedImage=DatasetType(StorageClass=MaskedImage, DataUnits=("Visit", "Sensor")))
# registry.registerDatasetType(CalExp.image=DatasetType(StorageClass=Image, DataUnits=("Visit", "Sensor")))
# registry.registerDatasetType(CalExp.variance=DatasetType(StorageClass=Variance, DataUnits=("Visit", "Sensor")))
As we'll see below, these component DatasetTypes will be used by virtual components of concrete composites and concrete components of immediate virtual composites, but will not be used for components of deferred virtual composites (because those will have already been written and added to the Registry using some other DatasetType).
Datastore/Butler
To configure a composite DatasetType X to be written (put) as a concrete composite with virtual components:
- Provide a write formatter for X itself.
- Provide a read formatter for X itself.
- Provide a read formatter for each X.<component> DatasetType that reads just that component from a file written using the write formatter for X.
- Write formatters for X.<component> must not be provided.
To configure a composite DatasetType X to be written (put) as an immediate virtual composite:
- Provide an assembler for X.
- Provide a disassembler for X.
- Provide a read and write formatter for each X.<component> DatasetType that read/write that component from/to its own file.
To configure a composite DatasetType X to be written (linked) as a deferred virtual composite:
- Provide an assembler for X.
Writing Datasets
Pseudocode, no error-handling/transactions:
def Butler.put(self, obj, datasetType, dataId, producer=None):
"""Write a dataset.
May not be a virtual component or a deferred virtual composite.
"""
datasetType = self.registry.getDatasetType(datasetType) # argument may have just been a string; now it's an object
ref = self.registry.addDataset(datasetType, dataId, run=self.run, producer=producer)
disassembler = self.config.getDisassembler(datasetType)
if disassembler is not None: # this is an immediate virtual composite
childObjs = disassembler(obj)
for childName, childDatasetType in datasetType.components.items():
if self.config.getWriteFormatter(childDatasetType): # not a virtual component
childRef = self.put(childObj, childDatasetType, dataId, producer=producer)
self.registry.attachComponent(parent=ref, child=childRef)
self.registry.setAssembler(ref, self.config.getAssembler(datasetType)) # Could also consider putting this in Datastore
else: # this is concrete (and maybe a composite)
for childName, childDatasetType in datasetType.components.items(): # if not a composite, loop body is never executed
childRef = self.registry.addDataset(childDatasetType, dataId, run=self.run, producer=producer)
self.registry.attachComponent(parent=ref, child=childRef)
self.datastore.addReader(childRef, self.config.getReadFormatter(childDatasetType))
self.datastore.put(obj, ref)
return ref
def Butler.link(self, datasetType, childRefs, dataId, producer=None):
"""Create a deferred virtual composite dataset by associating existing datasets.
There are two link overloads; this one is more powerful but less convenient in the common case.
"""
ref = self.registry.addDataset(datasetType, dataId, run=self.run, producer=producer)
for childRef in childRefs:
self.registry.attachComponent(ref, childRef)
self.registry.setAssembler(ref, self.config.getAssembler(datasetType))
def Butler.link(self, datasetType, childDatasetTypes, dataId, producer=None):
"""Create a deferred virtual composite dataset by associating existing datasets.
There are two link overloads; this one is less powerful but more convenient in the common case.
"""
# Look up the DatasetRefs using the given DataID and then call the other overload.
self.link(datasetType,
[self.registry.find(childDatasetType, dataId) for childDatasetType in childDatasetTypes],
dataId)
def Registry.addDataset(self, datasetType, dataId, run, producer=None):
dataset_id, registry_id = self.execute("INSERT INTO Dataset ...")
return DatasetRef(datasetType, dataId, dataset_id, registry_id, ...)
def Registry.attachComponent(self, parent, child):
self.execute("INSERT INTO DatasetComposition (parent_dataset_id, parent_registry_id, component_name, child_dataset_id, child_registry_id) ...")
def Registry.setAssembler(self, ref, assembler):
self.execute("UPDATE Dataset SET assembler=? WHERE dataset_id=? AND registry_id=?", assembler.name, ref.dataset_id, ref.registry_id)
def Datastore.put(self, obj, ref):
# ... actually write a file or something ...
self.registry.execute("INSERT INTO Storage (dataset_id, registry_id, datastore_name, md4, size) ...")
self.addReader(ref, self.config.getReadFormatter(ref.datasetType))
def Datastore.addReader(self, ref, formatter):
# ... record somewhere that we should use the given formatter when asked to read back ref ...
Reading Datasets
Again, just pseudocode, no error handling:
def Butler.get(self, datasetType, dataId):
ref = self.registry.find(datasetType, dataId)
if ref.assembler is not None: # virtual composite
return ref.assembler({childName: self.datastore.get(childRef.dataset_id, childRef.registry_id)
for childName, childRef in ref.components.items()})
else: # virtual component or concrete
return self.datastore.get(ref.dataset_id, ref.registry_id)
def Registry.find(self, datasetType, dataId):
ref = DatasetRef(self.execute("SELECT * FROM Dataset WHERE dataset_type_name=? AND ...", datasetType.name, dataId))
if ref is None:
# If this is a component of a deferred virtual composite, the Dataset table will have its original
# dataset_type_name and data ID, and the above query will fail. Instead we look for that composite
# and then find the component by name.
parentDatasetTypeName, componentName = datasetType.split() # split the part before the first "." from the part after it
datasetEntry = self.execute("""
SELECT Child.* FROM Dataset AS Child INNER JOIN DatasetComposition ON (...) INNER JOIN Dataset AS Parent ON (...)
WHERE DatasetComposition.component_name=? AND Parent.dataset_type_name=? AND Parent.{data_unit_fields}=...
""",
componentName, parentDatasetTypeName, dataId
)
if datasetType.components:
ref.components = {
result["name"]: Dataset(result)
for result in self.execute("""
SELECT Child.* FROM Dataset AS Child INNER JOIN DatasetComposition ON (...)
WHERE DatasetComposition.parent_dataset_id=? AND DatasetComposition.parent_registry_id=?
""",
ref.dataset_id, ref.registry_id
)
}
return ref