...
- There is no camera-generic way to express a concept like "visit", even thoug though grouping by such concepts is frequently necessary in pipeline code.
- Each camera must provide its own path template for any dataset that includes data ID keys related to observational data, making the definition of new DatasetTypes much heavier.
...
Pipelines that create master calibration products frequently use the reverse of this relation between Exposures and ExposureRanges: they take as inputs raw datasets identfied with Exposure DataUnits (e.g. raw flat observations) and produce, as outputs, datasets identfied identified with ExposureRange DataUnits (as well as various instrumental DataUnits, of course). It will also be possible to provide a custom mapping from input Exposures to output ExposureRanges by providing configuring a SuperTask with custom SQL (details still TBD).
...
Because SuperTask Pipelines can in general process data from multiple cameras, the Gen2 approach of automatically applying Camera-specific configuration overrides doesn't make really make sense. The (vague) approach I have in mind for Gen3 is for each obs_* package to also define one or more Pipelines, ideally by expressing the diff against some fiducial definition of a Pipeline. This could involve more than just configuration changes; it could also involve adding or removing entire SuperTasks. These Camera-specific Pipeline definitions would be passed directly or indirectly to the SuperTask execution driver code (where additional command-line overrides could be applied).
A Camera-specific Pipeline would still, of course, only be appropriate for data from that Camera, and I expect us to continue to predominantly process data from only one Camera at a time. This approach leaves room, however, for multi-Camera Pipelines to be defined in other packages (though these may be limited to just those steps that can be accomplished with the same configuration for all Cameras). It's also worth noting that this approach doesn't provide a way to stop users from accidentally using using a Pipeline configured for one Camera with data from another.
...