John Swinbank: Ensure pipelines code for regenerating images on demand is included in LDM-151, that an equivalent service is included in the appropriate DAX design documentation, and that both of them are included in the DM construction plan.
Fritz Mueller: Develop a complete (to the best of our current understanding) list of data products that have to be accessible through DAX services, and a plan for extending that list in future.
Colin Slater: Publish technical note on next-to-database processing use cases.
Wil O'Mullane: Following the publication of Colin Slater's technical note on next-to-database processing, decide what further meetings or discussions are required.
There is currently a lot of ad-hoc, grass-roots effort going into cleaning up the stack and its documentation that has at best inconsistent support from the DMLT, and this is both distracting developers from scheduled work and leaving them frustrated because the effort they're donating via free/science time isn't making a big dent. DMLT needs to agree to and present a united and coherent policy and plan for modernization so we aren't fighting our own devs (and each other) on this. I will come with at least one rough proposal, but it may be too ambitious to be tenable, and I'd really like other members of DMLT to come armed with additional proposals
Please come to this discussion prepared as Jim suggests.
Frossie Economou & Jim Bosch: Organize a three day “retreat” to discuss Pipelines codebase modernization.
17:00
Close
18:00
Everybody is invited to join the DM Project Manager & the local UW LSST team for drinks at Ravenna Brewing, 5408 26th Ave NE (25 minutes walk from the meeting venue, 15 minutes by bus).
19:30
Dinner at BaBar, 2685 NE 46th St (10 minutes walk from Ravenna Brewing).
If you will be coming for dinner, please provide Wil O'Mullane with $34 (your dinner per diem).
If you will not be coming for dinner, please let John Swinbank know in advance so we don't wait for you.
The restaurant will accommodate all the dietary needs that we know of; please make your needs known to the server.
Fritz Mueller: Provide example QuantumGraph to the Data Facility. Initial example (illustrative of QG API) was sent. Andy S. awaiting feedback/questions about the the API from NCSA.
Portal, Firefly, Qserv, Header Service, "L1" Image Ingest, Workload/Workflow, Data Backbone, Jellybean, SQuaSH: these use different languages, different dependencies, different skills than Science Pipelines, Task Framework, Butler. Their code is generally not reviewed, understood, or maintained outside of their own groups. What standards and processes are we and should we be applying to these to ensure that they function well and are maintainable for the lifetime of the project?
Kian-Tat Lim: Figure out which (project and/or DM level) documents need to be updated to better capture standards & processes for development within DM and elsewhere.
Currently, the standard processing results in FITS files containing the afwTable catalogs from the processing. These are then transformed to Apache parquet format for use in QA endeavors. This means we are storing the same information in two redundant formats. I would like to explore how we treat the various file formats.
Do we treat some as transient and others as persistent?
Can we settle on a single format that will serve multiple needs with the understanding that some transformation may be necessary to serve the community?
Depending on the above, are there impacts on the sizing model that need to be taken into account (realizing that the sizing model is specifically for operations)?
Based on the different documents available (and in preparation), I would like to share some thoughts, maybe a proposal, for a categorization of the different products that can be identified (at the moment) inside DMS. (GCM_ProductTree.pptx)
Metadata (units, UCDs, descriptions, linkages, etc.) by the pipelines
Nightly reporting
The relationship between DBB and LSP user file storage space. This was brought up at the LSP workshop December 5, 2018. Several of us think there may need to be a working group for this.
Topic
Requested by
Time required (estimate)
Notes
Update on status of Summit - Base ITC and Networks
There is currently a lot of ad-hoc, grass-roots effort going into cleaning up the stack and its documentation that has at best inconsistent support from the DMLT, and this is both distracting developers from scheduled work and leaving them frustrated because the effort they're donating via free/science time isn't making a big dent. DMLT needs to agree to and present a united and coherent policy and plan for modernization so we aren't fighting our own devs (and each other) on this. I will come with at least one rough proposal, but it may be too ambitious to be tenable, and I'd really like other members of DMLT to come armed with additional proposals.
Look at JIRA test director and how SE are using it. Also how might we use this to make our test specs, test reports and traceability. There is also some terminology difference to be covered (test spec = test run etc. ) .
Based on the different documents available (and in preparation), I would like to share some thoughts, maybe a proposal, for a categorization of the different products that can be identified (at the moment) inside DMS.