Note that the “Dinky” — the train from Princeton Junction to Princeton — will be suspended during the DMLT meeting. A substitute bus service will be provided by NJ Transit. Please allow extra time for your journey.
We discussed whether some DM work can be pushed into the future as effectively a “no cost extension” to construction; the practicalities of this are not clear...
In terms of what's acceptable to the agencies.
And in terms of whether there will be staffing in the operations era to work on this.
If there are people in your team/institute who would be interested in moving to AURA positions, please let Wil O'Mullane know.
There are practical issues here, regarding which states are acceptable for people to reside, and whether there is office space available for them.
And of course the downsides to breaking up groups.
DM10 alert filters interface may actually represent a scope increase in 02C.03 in terms of providing a mini-broker interface.
We believe that the current portal is sufficient for commissioning data releases (after finishing the closeout plan).
We discussed increase staffing load resulting from making commissioning data widely available: this is expected to be significant.
“As discussed at the PST F2F meeting in September, DM should review the names of people going into commissioning” (requested by Leanne Guy )
Not immediately clear whether this means permanent reassignment of staff, or those DM folks who are temporarily assigned to assist the Commissioning Team with specific activities.
The commissioning support activities listed are validation of the DM system, but verificationof the LSST system.
None of the commissioning activities listed cover aspects of the system outside Science Pipelines.
Concern expressed about the impacts of people being reassigned to commissioning on the rest of the DM team.
Often the people who might be most effective with commissioning are also those most necessary for facilitating efforts within DM; concern expressed that this will have a disproportionate impact on the DM schedule.
Detailed definition of the contents of the commissioning tests is with Leanne Guy and Keith Bechtol . Expectation that they will often involve repeating tests that have been performed within DM.
Early Ops funding is now available, and during FY19 (ie, this year) the ADs for both Data Facility and Science Operations (Margaret Gelman and Wil O'Mullane) are funded at 0.25 FTE. That's half an FTE coming out of DM management. How are we handling that?
Similarly there's a total of ~15 FTEs funded across Data Facility and Science Ops during FY20.
Please review:
The plans and schedule for transitioning staff;
The activities which the Operations Team will be carrying out with this effort, and how they relate to ongoing DM construction and the commissioning effort.
Some milestones (e.g. ops rehearsals) are effectively duplicated between DM and pre-operations funding; where possible, they will migrate from DM to pre-ops.
There's a worry that commissioning data maybecome available to the public more quickly than we currently plan; DM should be ready to scale up to address this.
And a feeling that simply making the data available for download will not adequately address this need.
But we acknowledge the cost and schedule impacts of this.
There's also a concern about what level of end-user support is implied by this.
How do we handle folks being required in DM and commissioning and pre-ops?
This is a matter of ongoing planning. There may be some overlap/double counting.
Discussion of how open the access to commissioning data & facilities should be, balancing getting input on commissioning from members of the community with the support load and “chaos” of wide access.
Wil O'Mullane — coordinate the writing of a memo describing what community DM can support during commissioning.
The proposal is to develop a policy for handling packages which have been developed externally and which their authors offer up for inclusion in pipeline processing, the Science Platform environment, or elsewhere in the DM system.
Obvious examples might be scientific algorithms contributed by the community.
There is history of external users refusing to make contributions like this due to the demands of DM engineering (code quality, review, tests, etc...).
Based on the modeling work done before the review this year, review proposed product tree, components characterization and relation with the document tree
“Inside pipelines, there are more or less five products” — not every Git repository or software package is a product.
“SW Products can depend on other SW products without containing them” — so there is a SW product that contains e.g. the Butler, which can be depended upon by other products.
SW Products are the unit of release.
Dependency relationships happen between SW products, rather than between SW packages.
All DMLT: Review product tree in LDM-294 and provide feedback/corrections to Unknown User (gcomoretto) .
The Data Facility can execute based on whatever version of the middleware the developers are using.
There's no requirement for pipeline developers to support Gen2 from the LDF point of view.
Most Data Facility work going forward is based on Pegasus; there is minimal ongoing support for DESDM.
Assuming availability of pipeline code, the LDF predicts that they could run pipelines in a “sustained processing” mode based on Butler G3 and Pegasus in mid-2019.
Two possible goals for BG3 priority:
Support for obs_lsst (to make RHL & Merlin's life easier)
Or to convert code to PipelineTask to enable execution at scale of e.g. HSC on the Data Facility.
The consensus is that the latter is the priority; agreed to prioritise the conversion of ci_hsc Tasks to PipelineTasks until end of Jan 2019 per Fritz Mueller 's recommendation.
This would also meet Frossie Economou 's immediate validate_drp use case.
We agreed that temporarily abandoning the shared-nothing model for execution might enable faster development.
We should also prepare for “plan B” by assembling a “mini-working-group” to consider wholesale technological change (mini-WG to consist of at least Kian-Tat Lim & Simon Krughoff; not to involve folks who are busy with the ci_hsc conversion ).
Kian-Tat Lim , Simon Krughoff — convene a mini-WG to create an alternative plan to continuing with the existing approach to PipelineTask/ButlerG3 middleware.
John Swinbank — designate and/or start a recruitment process for a “systems programmer” to act as long term middleware owner.
Are there ways of providing greater uptime and lower latency using rolling upgrades, schema evolution with backfill, planned Observatory maintenance windows and daytime maintenance, etc.?
We're lacking L3 milestones in PMCS which describe the availabilities of services and capabilities which we know are coming, particularly later in construction.
Obvious examples include:
Butler Gen 3 / PipelineTask as the regular production environment;
Roll-out of the Pegasus WMS (or some other WMS);
Data back bone capabilities;
There are no DAX milestones beyond the end of November 2018;
There are five SQuaRE milestones total, all of which refer to notebooks (and which are not necessarily SQuaRE deliverables — see below).
In addition, some existing milestones seem unclear about who is delivering what. For example:
“DM-SUIT-16: Commissioning DAC“ — is that really a SUIT deliverable?
“DM-SQRE-5: Notebook service ready for general science“ — is that really a SQuaRE deliverable?
I assert:
We should have milestones describing the delivery of effectively everything described in LDM-148.
Are there other things which are not in LDM-148 which we need to track?
“Standing up a service” milestones are all for the Data Facility, and should be dependent on prerequisite milestones for software delivery.
We should have brief (~few sentence) description for every milestone.
We might consider having a more formal verification procedure, which would relate successful completion of these milestones to test execution.
I suspect that such a procedure would lose us more in time & overheads than it gains us, but I am open to being convinced otherwise.
Can we produce a revised set of milestones? When is an appropriate due date — January 2019?
Are there ways of providing greater uptime and lower latency using rolling upgrades, schema evolution with backfill, planned Observatory maintenance windows and daytime maintenance, etc.?
Work has been ongoing in the DM-SST to develop a formal DM data model and to refine the way in which LSE-163 (DPDD), the cat pacakge, LDM-153 (the baseline schema) are generated and managed.
This should be presented to and agreed by the DMLT.
Early Ops funding is now available, and during FY19 (ie, this year) the ADs for both Data Facility and Science Operations (Margaret Gelman and Wil O'Mullane) are funded at 0.25 FTE. That's half an FTE coming out of DM management. How are we handling that?
Similarly there's a total of ~15 FTEs funded across Data Facility and Science Ops during FY20.
What are the plans and schedule for transitioning staff?
Based on the modeling work done before the review this year, review proposed product tree, components characterization and relation with the document tree
We're lacking L3 milestones in PMCS which describe the availabilities of services and capabilities which we know are coming, particularly later in construction.
Obvious examples include:
Butler Gen 3 / PipelineTask as the regular production environment;
Roll-out of the Pegasus WMS (or some other WMS);
Data back bone capabilities.
In addition, some existing milestones seem unclear about who is delivering what. For example:
“DM-SUIT-16: Commissioning DAC“ — is that really a SUIT deliverable?
“DM-SQRE-5: Notebook service ready for general science“ — is that really a SQuaRE deliverable?
We should have milestones describing the delivery of effectively everything in LDM-148.
The half hour time estimate is for Wil O'Mullane to state the above, set expectations, and develop a timeline for delivery. If we actually start creating milestones during the meeting, it could take arbitrarily long.
The proposal is to develop a policy for handling packages which have been developed externally and which their authors offer up for inclusion in pipeline processing, the Science Platform environment, or elsewhere in the DM system.
Obvious examples might be scientific algorithms contributed by the community.
There is history of external users refusing to make contributions like this due to the demands of DM engineering (code quality, review, tests, etc...)
The aim is to share some preliminary considerations on the release process, taking in account the actual approach and giving a look on how this can evolve.
✅ (albeit in a potentially tightly squeezed slot...)