This page has been created to gather feature requests for the Generation 3 Butler that don't obviously make sense as capital-R Requirements or JIRA tickets.  These can be Gen2 Butler mistakes that you want to make sure we don't repeat, thoughts on how to make the interface more user-friendly, and micro use cases that are well below the level of detail of the officials.

To add an item, please edit this page and add "task list" item to the list below, so we can use Confluence actions to check it off.  Please include your name with an @-mention; this will mean that the Task is formally assigned to you, which isn't quite ideal, but we need some way to make sure we can follow up on anything posted here, and that will also make it easy for you to follow the status of the item.

  • Ensure that NumPy scalar types can be used in data IDs transparently (from John Parejko).
  • _filename magic should resolve to actual path of file (from Eli Rykoff)
  • Be able to iterate over all possible Dataset Types for a given DataId (set of Data Units) in a given Collection, in addition to the more common iteration over all possible DataIds for a given Dataset Type (from gpdf  via KTL)
  • CameraMapper.yaml description fields (a description of what the dataset is) that get lifted into the instantiated butler, so one can say butler.describe('calexp') (or something similar) and get the description of that dataset. (from John Parejko).

The Gen3 middleware team will periodically scan these items and check them off with an edit to explain why.  This could be translating the request into a JIRA issue, actually implementing it in the prototype as part of another ticket, or upgrading it into a formal Requirement or Use Case (we will frequently ask the author to help with those steps).

Operations related items

The following are various things that Operations will want to do (most with the Data Backbone, but others with the job sandbox).    These may not be features that get built into Gen3 Butler, but may be external tools that use information saved in the new Registry/DataStore tables or require specialized Registry or DataStore codes.

There were the much longer Operations Use Case descriptions from the beginning of the Butler Working Group (Operations Use Cases, Operations Use Cases 2). 

As the design of the Registry and DataStore have emerged, here are some more things that are related.   There are also some more items as some provenance is being stored for DataSets.


  • Chained DataStores would implement any functionality by iterating over all of its sub-DataStores.    If this is not the case, we should point that out.
  • If absolutely necessary, one could pre-create a Collection (tag) using more flexible queries and the tag can be used in the following items.
  • For some combination of DataSetTypes, Collections, Run ids, and other DB tables, do they actually exist in a given DataStore.   (For large DataStores like the Data Backbone, it would be enough to say that the file tracking information stored in DataStore(?) tables is good enough to answer this question.)
  • For some combination of DataSetTypes, Collections, Run ids, and other DB tables, how much physical/disk space is being used by a DataStore.
  • Delete files from a given DataStore for some combination of DataSetTypes, Collections, Run ids, and other DB tables.
  • Validate DataStore
    • Check that some set of DataSets that are supposed to be in DataStore actually exist (match tracking entries in DB to files on disk)
    • Check that there are not extra files on disk that are not tracked by DataStore
    • For some combination of DataSetTypes, Collections, Run ids, and maybe other DB tables, check that the files are valid (i.e., correct file size, correct chksum)
  • Ability to check that saved correct information in Registry/DataStore for runs.    Ability to compare Registry/DataStore type information across runs.
  • (provenance data related)  How many times did a particular SuperTask execute during a run/pipeline execution?    (How many times did each SuperTask execute during a pipeline execution?)
  • (provenance data related)  Run queries to check for non-science issues such as "bad" machine (all jobs fail on machine), dramatic increase in run times, memory, failures, etc.
  • If Operations divides a pipeline into sub-pipelines, need ability to have the "run id" be the same throughout (e.g., assume completely successful run where a pipeline is divided into sub-pipelines to enable mid-run pre-flights steps).
  • No labels