Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


Day 2, Wednesday

Moderator: Wil O'Mullane 

 Notetaker: Yusra AlSayyad 


Status and plans:








DM Science

>>> presentors = ['Leanne', 'Cristian', 'Frossie', 'KT', 'Fritz', 'Stephen', 'Ian', 'Yusra']

>>> random.shuffle(presentors)

>>> presenters.remove('Fritz')
>>> presenters.append('Fritz')


['Fritz', 'Frossie', 'Yusra', 'Cristian', 'KT', 'Ian', 'Stephen', 'Leanne']

DAX F22/S23 Status_Plans.pdf

Cristian Silva - DMLT vF2F October 2022.pdf

Arch F23A Status and Plans.pdf

DM Science Retrospective and Plans F22AB

NCSA DMLT F2F Oct 2022.pdf

  • K-T: Is it possible to pay CADC?
  • Frossie: No, but it is possible to trade some of our systems with some other systems. 

  • YA: Been announcing since March, but if you didn’t get the memo with Scarlet Lite’s computational performance improvements, we’re not worried about having resources to run it.
  • Wil: Does this mean we can retire that risk?
  • YA: No, because that risk was about ALL algorithms, and just because we’re not worried about Scarlet anymore doesn’t mean that we’re not worried about things like galaxy measurement. We still have more algos than we have compute for, and there will be hard tradeoffs to be made. 

Wil: And Andres is going to give a talk in Spanish for us next week. 

  • Colin: The backup link, that’s microwave from the summit to where?
  • Cristian: Just summit to base. It’ll be invisible. When the fiber goes down, we change the priority to wireless. It’s only for control data, not science data. 

  • Wil: We had a network meeting with SLAC last week. Are they OK with the routers?
  • Cristian: Yep. We need a list of what needs to be reachable from Rubin.
  • Wil: The question from (didn't catch the name?) about the suitability of the routers for installation?
  • Cristian: Oh, we didn’t get to that. We gave him the specs. It’s a router behind our routers at the summit. Another at SLAC. one for the 100, one for the 40
    •  Richard Dubois Confirm on your side that SLAC is OK with the router specs?  

Wil: Cristian, we should talk with Kevin and give him a delivery time so that he can put in P6 a date that it will be in place. Then we can communicate with NSF about something happening. We have a milestone but need an activity. 

  • Wil: Good about the Postgres. Tiago was showing me this serverless Postgres he set up, and I told him that you have something like that with Kubernetes that he can use.
  • Richard: Do you need help from the Fermilab DBA folks to set up the Postgres on Kubernetes?
  • Cristian. Sure! The servers are already running. Who do I write?
  • Richard: <not taker didn’t catch the name>.
  • Wil: We’ll want the help once everyone is adding their schemas.
  • K-T: to the extent that they are using the butler, it’s just one schema. I think it’s up to Steve and me to migrate the SQlite DB for the OODS over to Postgres. Got bogged down.
  • Steve: Yeah, I’m in contact with Bruno about that. 

  • Frossie: K-T, What was the metrics thing?
  • K-T: Thinking about how to get metrics into Sasquatch. To my knowledge, the way it’s done down is outside of the pipeline.
  • Frossie: Yeah, but Angelo is working on it now. 

  • Richard: Jenkins workers running in the cloud… what’s the need to move it to SLAC?
  • K-T: no requirement, but it would allow us to do things Merlin asked for. He wants to run pipeline tests against live butler repos. We never had it at NCSA, even though we could have.
  • Frossie: Those jobs were run at the DF so that we could run deep tests with the production system.
  • K-T depends on how much we want to own everything
  • Cristian: Is LFA replication bi-directional?
  • K-T: No
  • Cristian: we could use our multi-site replication!
  • Wil: Let’s punt this to a future discussion. 
      •   Kian-Tat Lim  Convene a meeting following up vF2F discussion about what to do with Jenkins  

Moderator: Wil O'Mullane Notetaker: Ian Sullivan 

11:00Needs for "tactical" databasesRobert Lupton See RHL's rough notes
  • FE: There are many producers and many consumers of metrics. Each has different needs
    • Elephant in the room is the consolidated database. Metrics were going to be one of the things persisted with the Butler
    • Stuff that needs to go from USDF to the summit needs to go through Sasquatch
    • Stuff that needs to be presented to the user needs to be in the consolidated database
  • RHL: thinks there could be other ways of moving stuff from USDF to the summit
    • this is a proposal for what we need, which might be what the consolidated database may be
  • FE: We have tooling other than Chronograf on the summit, also have notebooks
  • KTL: Rucio is another method to transfer (calibration) datasets from the USDF to the summit
    • Any time series-only data should go in the database
  • JB: interested in looking into the database keys that are explicitly time oriented.
    • WOM: should follow up outside this meeting
  • FE: Concern about camera data that is not in DM-land.
  • RHL: Interested in implementation here. The camera team has tools, how do we use them?
    • WOM: We can port the tools, and make the queries talk to influx DB.
  • FE: Kafka plus influx is Sasquatch. Is it true that camera telemetry scalars are going in the EFD?
    • RHL: Almost
    • WOM: will be down there soon, and can assist with the port if needed.
  • KTL: This mostly deals with the keys, not the values. Those will change over time, and will require talking with the scientists
  • FE: I don’t see how this schema gets defined without GPDF. We need data engineers
  • GPDF: I don't need to be involved in the definition of each table, but yes to deciding what keys are time varying etc.
  • WOM: Once we have postgres at the summit, we should start putting schemas in it
  • JB: What kind of skills does the person setting up the schema at the summit need to have?
    • WOM: that is what I am trying to work out.
    • CS: The need is for someone to understand the conditions at the summit
    • WOM: That's Patrick, Tiago, Erik, Robert
      • Question is whether we can make use of general knowledge of people available at Fermilab
  •  Kian-Tat Lim Determine whether people at Fermilab can assist with setting up schemas for the summit  
  •  Robert Lupton Identify point of contact on commissioning side for summit schema  
  •  Robert Lupton Kian-Tat Lim Kick off meeting between commissioning, Fermilab, and summit (Carlos) for summit schema  

11:30 (DMLT only)Wrap Up/Actions/AOB/
  • YA: Campaign management reorganization (slides)

      • LG: Why are Colin and Yusra interim product owner and group lead?
        • YA: if someone signs up for the campaign management team, I want to leave these roles open to them.
        • LG: might be good to write current, not interim
        • YA: hoping that someone will step up and want to do the group lead role. Agree current is a better term
      • KT: pilot/co-pilot rotation sounds like a great idea, but people are not necessarily replaceable in what they were working on before. Is it expected that people will pause other work, and it will be restarted when they return from campaign management?
      • YA: I don't want people to feel trapped in this role, which can easily happen.
      • RD: Gotta run. We discussed this at the retreat so I’m ok with it. Remember there can be members from Europe. We also worried that 100% would indeed burn people out.
      • EB: There may be distinctions between how this looks for DRP/AP/commissioning.
        • WOM: Frossie has some org charts, we will write it up with more detail
      • GPDF: very supportive. Looks a lot like the mechanisms we set up eventually with BABAR. 100% agree this tends to burn people out; there is a small subset of people who enjoy it.
        • agree with EB that there will be significant differences in rhythm between AP and DRP
      • LG: question for Colin:
        • do you see different people in V&V rotating through as well?
        • CS: This is not a position I intent to sit in to hold it, would be happy for someone else to step up as product owner, but it is too important to leave empty at first
        • LG: I think it would be very helpful to have V&V team members rotate through, to spread experience.
          • WOM: really like the idea of more people having experience organizing/leading. Builds resiliance
      • FE (slides)
        • This plan that YA proposed came out of DPP retreat last month.
          • Campaign management now part of Data Production under YA, next to Algorithms&Pipelines
          • Qserv now in Data Services under FE
  • FE: format of this meeting
    • Propose that we upload status slides in advance, use the timeslot for questions and discussion (standup style)
    •  FM: finds it useful to write/present plan in response to the discussions from DMLT, would find it less useful to prepare in advance
    • JB: slides written to be read are different than ones written for presentation, since that allows greater clarification
    • WOM: We used to reserve some time at the end for discussion
    • WOM: suggestion is to start last session later, so that people have time to finish their presentations. Provide drafts in advance. Provide time for reading the presentations, and schedule time (~1hr) for discussion afterwards.