Logistics
Date
–
Location
This meeting will be virtual; details of the teleconferencing system are to be determined.
Join Zoom Meeting
https://illinois.zoom.us/j/8154193943?pwd=N0JPZTV1RFZsY1dwUWMxa0E4N0RjQT09
Meeting ID: 815 419 3943
Password: 131840
One tap mobile
+13126266799,,8154193943# US (Chicago)
+13017158592,,8154193943# US (Washington D.C)
Attendees
- Wil O'Mullane
- Robert Lupton
- Tim Jenness
- Frossie Economou
- Yusra AlSayyad
- Jim Bosch
- Leanne Guy
- Unknown User (mbutler)
- Colin Slater
- Robert Gruendl
- Kian-Tat Lim
- Fritz Mueller
- Gregory Dubois-Felsmann
- Unknown User (gcomoretto)
- Ian Sullivan
- Unknown User (npease)
- Simon Krughoff
- Zeljko Ivezic
Agenda
Day 1: 2020-11-17 | ||||
---|---|---|---|---|
Time (Project) | Topic | Coordinator | Pre-meeting notes | Running notes |
Moderator: Yusra AlSayyad | Notetaker: Kian-Tat Lim | |||
09:00 | Welcome | Wil O'Mullane |
| Past actions:
|
9:15 | Project news and updates | Wil O'Mullane |
| |
9:45 | Plan to end of project: Part I | Wil O'Mullane | Victor will want a plan to the end of construction
Look at burn rates — how far do we get with the money we have?
Transition of personnel to Operations needs to be checked
Do remaining milestones make sense? Do we have effort to achieve them? When are things operational?
Justifying COVID expense:
Presented spreadsheets showing burn rates and needs through end of FY23 USDF starts as Ops when Ops starts
Plan for tomorrow:
| |
10:30 | Break | |||
Moderator: Kian-Tat Lim | Notetaker: Yusra AlSayyad | |||
11:00 | Plans for DM serving photometric redshifts | Leanne Guy | Leanne Guy will give an overview of the current status of the plans and then open the floor for feedback, input and discussion | RHL: How does proposed validation from algorithm candidates map to phase 3 when we do validation? KT: If an author team proposes a photoz and we don't select it, is it still an in-kind contribution? |
11:30 | Gen3 | We are in a transitional period. Gen3 just released to science pipelines; and awaiting feedback from users before deciding where to focus new development. In the meantime: potential discussion topics are:
| Test Plan Discussion: RG: We don't have a test plan: part my fault, part testing framework. Some tests are running now. Monika is running RC2. There was a request that she run all tracts together, but if you give it multiple things to run, the batch system will behave the same terrible way that the DES had: If one exposure has a problem, the job halts, and you have to work out what happened and restart. Test plans: Where do you balance modularity in the test plan with just getting done and being done with it?? "I can ingest a comcam exposure" "I can ingest an auxtel exposure." etc.. Do I write a test plan for each one? In the ops rehearsal, I wrote out each step of the test serially. You can't rerun it ever again. WO: Re mechanics of testing, worth having a chat with Jeff Carlin and Leanne. This might be one-off, though. RG: It'd be worth writing down: this is how you ingest a raw comcam exposure. "Go get a raw exp, and do an ingest. check No/Yes" This is not the right way to do this. I'm authoring it. Executing it, and arguing why what I did was fine. GDPF: What RG says about "self dealing" has been the exact same for the science platform. It'd be nice if we had someone with an independent point of view. TJ: I was expecting this to be more collaborative with Science Pipelines. RG: No one else has time for that either. This is more important for telling external users that is fit for use. RHL: We can help with my integration work. The "does it ingest" is coming from the outside. JFB: One prob is we wrote a Big bang milestone for a gradual process. It's inching ahead. We just now declared that the schema is stable, and it is worth switching dev to daily work. The important part of the milestone is schema stability, and the rest is the box-ticking exercise. RG: This is one step more formal than the boxes on the confluence page. I'm not saying that I shouldn't be doing what I'm doing. TJ: We can declare that we're adopting DM-DAX-12 whenever we want. If it's a 503 we can't.
------- Jim Bosch's next steps (slides) Questions: TJ: When we mean pain points, we mean command lines being weird. That means usability. happy to make improvements RHL: Where do we stand on remote butler access or butler exports? CS: Are there other worries you haven't enumerated here re PP? RHL: We can't special case Alert processing. we should be using generic mechanisms. Task: Discuss. Someone has to write a Pipeline.yaml. If its writing a new pipetask for this, then we have to talk about it. RHL: There's a layer of controlling the processing, that we're not paying attention to. e.g. maybe we define the visits outside. KSK: RHL: How do we manage those lists. Who owns the job of this? Frossie is going to say that we cal use OWL but what DB does it talk to? I'm willing to define it to not be middleware, but someone has to own it. There are few tables Jim put in the butler for good and bad exps, but that doesn't cover it.
[We ran out of time for Jim's presentation of the RFC for user collections]
| |
12:30 | Break | |||
Moderator: Wil O'Mullane | Notetaker: | |||
13:00 | HIPS requirements | (Contingent on Gregory's not being needed at an IVOA session) | Have had requirements for some time, LCR added them to DMSR (DMS-REQ-329, 379-385) HiPS visualization was already added to Firefly (funded by LSST) MOC visualization mostly done (part LSST, part IRSA), but weakness around discovering MOCs New versions of Aladin Lite will take HiPS maps Producing HiPS:
Annotate HiPS with ids of underlying coadd patch images
Highest-resolution is data rights controlled
Discovery is iffy
We need to write DRP tickets for generating this or using hipsgen
Blended color version could be lower priority RobertG: PTIF has recoloring (but might not be hue-preserving) Jim: Interactions with healsparse
MOC:
Yusra: when do you need them from a DRP?
Jim: Get someone to try doing ourselves for a couple of days
Server should be pretty trivial Could partner with SPHEREx for some development, as SPHEREx will be generating HEALPix all-sky maps | |
13:30 | <Newly Open> | |||
14:30 | Close | |||
Day 2: 2020-11-18 | ||||
Moderator: Wil O'Mullane | Notetaker: Ian Sullivan | |||
9:00 | AP Integration | Eric Bellm | The AP team is now testing precursor datasets large enough to require real databases. Do we push forward with Postgres at NCSA? Try to integrate Cassandra? More broadly, can we discuss the path towards the DM-AP-16 ( Full integration of the Alert Production system within the operational environment) milestone, with a view towards commissioning and pre-operations activities? | SQLite - Postgres - Cassandra
All the other pieces for integrating AP
|
9:30 | New framework for metric computation | Motivation and current status | New framework for metric computation – Leanne and Simon
| |
10:00 | Plan to end of project: Part II | Milestones – Wil
DAX Plan to end of construction – Fritz
DAX estimate of effort to complete
| ||
10:30 | Break | |||
Moderator: Leanne Guy | Notetaker: Fritz Mueller | |||
11:00 | Plan to end of project: Part III | We can use Thursday as managers day if you want more time to go through in more detail. | SQuaRE (Frossie)
ARCH (K-T)
DRP/AP (Yusra)
NCSA (Michelle)
Wrap up discussion:
| |
12:30 | Break | |||
Moderator: Colin Slater | Notetaker: Eric Bellm | |||
13:00 | Team status |
| Yusra:
Michelle:
Frossie:
KT:
Leanne: Fritz: Ian:
Gregory:
| |
|
| We said we'd do Focus Friday provisionally until this Nov meeting. How's it going? Do we want to keep it up? |
| |
14:15 | Wrap up | next DMLTs:
| ||
14:30 | Close | |||
Day 3: 2020-11-19 | ||||
Moderator: | ||||
11:00 | Close |
Pre-Meeting Planning
Topic | Requested by | Time required (estimate) | Notes |
---|---|---|---|
Plan to end of project | 3x30 minutes | Perhaps we should have 3 separate sessions on planning to the end of the project some things to cover.
| |
Status report from the provenance working group | 30 mins | What is the status on this? | |
Status on plans for DM serving photometric redshifts | 1hr | Leanne Guy will give an overview of the current status of the plans and then open the floor for feedback, input and discussion | |
New framework for metric computation | 1hr | Leanne Guy and Simon Krughoff will present the new metric computation framework. | |
Next steps for Gen3? | We could discuss Robert Gruendl's test plan, Jim Bosch's proposal for repo organization of precursor data, Yusra AlSayyad's plans for pipeline conversion and expansion, and Tim Jenness's expectations on pipelines use and feedback. | ||
Focus Friday | 15 min | We said we'd do Focus Friday provisionally until this Nov meeting. How's it going? Do we want to keep it up? | |
Requirements for Science Pipelines to support HIPS images. | |||
AP Integration | 30 minutes | The AP team is now testing precursor datasets large enough to require real databases. Do we push forward with Postgres at NCSA? Try to integrate Cassandra? More broadly, can we discuss the path towards the DM-AP-16 ( Full integration of the Alert Production system within the operational environment) milestone, with a view towards commissioning and pre-operations activities? |
Attached Documents
Action Item Summary
Description Due date Assignee Task appears on 15 Mar 2022 Frossie Economou DM Leadership Team Virtual Face-to-Face Meeting, 2022-02-15 to 17 18 Mar 2022 Kian-Tat Lim DM Leadership Team Virtual Face-to-Face Meeting, 2022-02-15 to 17 17 Nov 2023 Frossie Economou DM Leadership Team Virtual Face-to-Face Meeting - 2023-Oct-24 30 Nov 2023 Yusra AlSayyad DM Leadership Team Virtual Face-to-Face Meeting - 2023-Oct-24 11 Dec 2023 Gregory Dubois-Felsmann DM Leadership Team Virtual Face-to-Face Meeting - 2023-Oct-24