Wil O'Mullane

Project news etc
----------------

- reminder to onboard new hires
- jira on cloud
  - needs co-ordination to ensure everybody's [inc syseng] plugins work
 

• Wil O'Mullane19 Oct 2022   to check with Ian for timelines on Jira Cloud

- Future DMLTs
  - spring DMLT at the La Serena JTM
  - avoid new moon for future meetings [rhl]
  - June 12-14
  - Oct 23-25
  
- chile JTM
  - reminder people need to arrive/early for summit trips

Fritz Mueller 

User-Generated Data Products.pdf

User-generated data products
----------------------------

- for temporary tables user only interacts with the TAP server [gpdf]
- for temporary tables ~10K rows [gpdf]
- temporary table plan is go
- persistent tables 
  - skyserver approach schema per use [wom]
  - will not let user specify schema order [gpdf]
  - Fritz Mueller  and Gregory Dubois-Felsmann  will outline an architecture in a technote

Cristián Silva 

Cristian Silva - DMLT vF2F October 2022.pdf

RHL: Initial spec was 2 cores per CCD (400 cores).

Right now not much being used out of 640 cores.

New nodes will arrive 7 months after order.

What is running where now?

• RSP, Sasquatch, and RubinTV run in yagan
• Control component CSCs run in yagan; Wil things this may be ~50 cores
• chonchon runs LFA
• amor runs LOVE

• Cristián Silva Get contributions to ITTN-014 to record everything running in yagan 21 Nov 2022 
  

Most heavy compute like full focal-plane wavefront will be done at USDF, but rapid analysis will be done in this cluster.

Power and cooling and space concerns? Far from space, close to limit of number of power sockets, but new UPS installed soon, so will have the ability to support more; limit is at least 1500 cores.

Anything that happens at the Summit must happen at USDF (or be transported there) as well in order to satisfy users who will want the same data products.

Can estimate based on number of parallel pipelines running in order to provide feedback to operators in near-realtime?  Maybe 2-4 such? (Therefore 400-800 cores.)  FAFF is defining these; they may be different than what is in Alert Production (e.g. PSF estimation).

Must distinguish between metrics that "must be there" (or will waste telescope time) and "mostly needs to be there" (can come from Alert Production).

USDF resources are also not particularly well-motivated.

Jira is also an issue for Summit independence.

Maybe start with an extra 400 to be safe, then adjust when we see what happens when ComCam goes on sky.

Wil O'Mullane 

Link to Data Facilties google doc for epics planning (and used in the meeting for adding more milestones)

Need to identify milestones that interact with CET, etc.

Epics are activities and milestones are points in time.

Only two milestones in DP right now:

• Ready for ComCam processing PREOPS-1379 - Getting issue details... STATUS
• Ready for DP1  PREOPS-1377 - Getting issue details... STATUS

Appropriate to have milestone for "Live image data+metadata from ComCam exposed to RSP so that it is available via ObsTAP" (work is in Middleware).  Can also add "DRP works at USDF" (blocks "Ready for ComCam"), "Alert Distribution works at USDF" (not a blocker for anything, equivalent to unclaimed Construction milestone).

A number of other USDF-based but cross-team milestones from Richard.

Distinguish between Ops work to operate systems and Ops work to build things not in Construction requirements; may affect which milestones to tie things to and whether to block milestones.

Science Pipelines Ops work PREOPS-1494 - Getting issue details... STATUS

• Porting to USDF
• Computational performance testing
• Analysis tools and RubinTV are "missing scope"
• Maintenance and evolution of delivered products (fgcm, piff)

Figuring out Ops processes (and documenting them) may be considered an Ops activity; should have a milestone for defining these for ComCam Operations.

Secure servers needed before end of 2023.

All USDF work is considered Ops.

Need to have a mechanism for caching of images at Cloud DF, but not an issue until DP2, and this is an optimization anyway.

• Frossie Economou Should have a test of how image access will work with Cloud DF using images from USDF, showing that Cloud DF is not significantly worse than going directly to USDF would be 31 Dec 2022 
• Wil O'Mullane to complete milestones in Jira 31 Oct 2022 

Wil O'Mullane 

Should not overlap meeting-free weeks with reviews, as those people need the weeks the most.

Perhaps alternate September week as Chilean holiday week (this year) with US Labor Day (2023-09-04).

April week the week of (2023-04-10).

June week the week of Juneteenth (2023-06-19).

• Wil O'Mullane 31 Oct 2022  Update dev guide to put in 2024 meeting free dates
  

Eric Bellm 

Forging ahead at USDF - no longer supporting GCP.

Tech selected - end to end demo not yet demostrated - ETA one month

AuxTel perhaps end of year.

Fritz Mueller 

Frossie Economou Yusra AlSayyad 

Cristián Silva 

Kian-Tat Lim 

Ian Sullivan 

Steve Pietrowicz 

Leanne Guy 

• K-T: Is it possible to pay CADC?
• Frossie: No, but it is possible to trade some of our systems with some other systems. 

• YA: Been announcing since March, but if you didn’t get the memo with Scarlet Lite’s computational performance improvements, we’re not worried about having resources to run it.
• Wil: Does this mean we can retire that risk?
• YA: No, because that risk was about ALL algorithms, and just because we’re not worried about Scarlet anymore doesn’t mean that we’re not worried about things like galaxy measurement. We still have more algos than we have compute for, and there will be hard tradeoffs to be made. 

Wil: And Andres is going to give a talk in Spanish for us next week. 

• Colin: The backup link, that’s microwave from the summit to where?
• Cristian: Just summit to base. It’ll be invisible. When the fiber goes down, we change the priority to wireless. It’s only for control data, not science data. 

• Wil: We had a network meeting with SLAC last week. Are they OK with the routers?
• Cristian: Yep. We need a list of what needs to be reachable from Rubin.
• Wil: The question from (didn't catch the name?) about the suitability of the routers for installation?
• Cristian: Oh, we didn’t get to that. We gave him the specs. It’s a router behind our routers at the summit. Another at SLAC. one for the 100, one for the 40
  • Richard Dubois Confirm on your side that SLAC is OK with the router specs? 03 Nov 2022 

Wil: Cristian, we should talk with Kevin and give him a delivery time so that he can put in P6 a date that it will be in place. Then we can communicate with NSF about something happening. We have a milestone but need an activity. 

• Wil: Good about the Postgres. Tiago was showing me this serverless Postgres he set up, and I told him that you have something like that with Kubernetes that he can use.
• Richard: Do you need help from the Fermilab DBA folks to set up the Postgres on Kubernetes?
• Cristian. Sure! The servers are already running. Who do I write?
• Richard: <not taker didn’t catch the name>.
• Wil: We’ll want the help once everyone is adding their schemas.
• K-T: to the extent that they are using the butler, it’s just one schema. I think it’s up to Steve and me to migrate the SQlite DB for the OODS over to Postgres. Got bogged down.
• Steve: Yeah, I’m in contact with Bruno about that. 

• Frossie: K-T, What was the metrics thing?
• K-T: Thinking about how to get metrics into Sasquatch. To my knowledge, the way it’s done down is outside of the pipeline.
• Frossie: Yeah, but Angelo is working on it now. 

• Richard: Jenkins workers running in the cloud… what’s the need to move it to SLAC?
• K-T: no requirement, but it would allow us to do things Merlin asked for. He wants to run pipeline tests against live butler repos. We never had it at NCSA, even though we could have.
• Frossie: Those jobs were run at the DF so that we could run deep tests with the production system.
• K-T depends on how much we want to own everything
• Cristian: Is LFA replication bi-directional?
• K-T: No
• Cristian: we could use our multi-site replication!
• Wil: Let’s punt this to a future discussion. 
  • •  Kian-Tat Lim  Convene a meeting following up vF2F discussion about what to do with Jenkins 24 Nov 2022 

• FE: There are many producers and many consumers of metrics. Each has different needs
  • Elephant in the room is the consolidated database. Metrics were going to be one of the things persisted with the Butler
  • Stuff that needs to go from USDF to the summit needs to go through Sasquatch
  • Stuff that needs to be presented to the user needs to be in the consolidated database
• RHL: thinks there could be other ways of moving stuff from USDF to the summit
  • this is a proposal for what we need, which might be what the consolidated database may be
• FE: We have tooling other than Chronograf on the summit, also have notebooks
• KTL: Rucio is another method to transfer (calibration) datasets from the USDF to the summit
  • Any time series-only data should go in the database
• JB: interested in looking into the database keys that are explicitly time oriented.
  • WOM: should follow up outside this meeting
• FE: Concern about camera data that is not in DM-land.
• RHL: Interested in implementation here. The camera team has tools, how do we use them?
  • WOM: We can port the tools, and make the queries talk to influx DB.
• FE: Kafka plus influx is Sasquatch. Is it true that camera telemetry scalars are going in the EFD?
  • RHL: Almost
  • WOM: will be down there soon, and can assist with the port if needed.
• KTL: This mostly deals with the keys, not the values. Those will change over time, and will require talking with the scientists
• FE: I don’t see how this schema gets defined without GPDF. We need data engineers
• GPDF: I don't need to be involved in the definition of each table, but yes to deciding what keys are time varying etc.
• WOM: Once we have postgres at the summit, we should start putting schemas in it
• JB: What kind of skills does the person setting up the schema at the summit need to have?
  • WOM: that is what I am trying to work out.
  • CS: The need is for someone to understand the conditions at the summit
  • WOM: That's Patrick, Tiago, Erik, Robert
    • Question is whether we can make use of general knowledge of people available at Fermilab

• Kian-Tat Lim Determine whether people at Fermilab can assist with setting up schemas for the summit 02 Nov 2022 
• Robert Lupton Identify point of contact on commissioning side for summit schema 26 Oct 2022 
• Robert Lupton Kian-Tat Lim Kick off meeting between commissioning, Fermilab, and summit (Carlos) for summit schema 09 Nov 2022