Skip to end of metadata
Go to start of metadata

Date

Attendees


Goals

  • Ensure successful use of the current NCSA infrastructure
  • Plan for near- and medium-term activities

Discussion items

TimeItemWhoNotes

Future meetingsUnknown User (jalt), Unknown User (pdomagala)
  • Paul will be taking over running current-infrastructure meetings with the next one, i.e., in two weeks time,  

Camera Test Stand 8 (TS8) data
  • RFC-301 - Getting issue details... STATUS covers the request to load this data.
  • Consensus seems to have been reached on the acceptability of incremental additions of data without additional RFCs
  • Discussion of the means of notifying NCSA staff that a new upload is complete and ready to be made mutable
  • Unknown User (pdomagala): would prefer that the request be transmitted as a JIRA ticket
    • Unknown User (pdomagala)  make a specific recommendation for the fields to use in these tickets (i.e., component, label, etc.) to ensure that they are seen by the right person and to facilitate summary queries

Hardware status
  • This morning's downtime is complete. All LSST GPFS filesystems now have the same block size.

HSC data
  • HSC PD1 (February public data release) has been loaded to /datasets/hsc/repo - see DM-9683 - Getting issue details... STATUS , RFC-297 - Getting issue details... STATUS

HSC processing
  • Lauren MacArthur and Hsin-Fang Chiang are collaborating on preparing to run the current versions of DRP code on HSC PD1.
    • It will be done with ctrl_pool
    • Still finalizing exactly what to run. Brief discussion of what to do about meas_mosaic. John Swinbank: it should not be run at this time. Later in the cycle we are likely to run a variety of tests of jointcal and meas_mosaic, and this will be looked at as part of QA work on the pipelines.
    • Planning to start run from latest weekly build but then hold version constant, if possible, for the whole dataset

Storage interfacesSimon Krughoff
  • Interested in learning more about the plans for data (file) storage, whether object stores will be used, how metadata queries can be turned into DataIDs usable in the Butler.
    • Relevant to some processing cases that don't obviously fit into the current DM pipeline paradigm: MOPS and alert distribution
    • Donald Petravick: current picture is that there need to be three databases involved in this area, enabling
      • Finding data by provenance
      • Finding data by metadata describing the file
      • Mapping data identity to data location (so that the "on-disk" arrangement of data can be changed)
    • Donald Petravick: working with Kian-Tat Lim to develop a process for working out requirements, initially, and then design, for provisioning for storage (as well as for computing); will present an introduction at Monday DMLT
      • Donald Petravick Make presentation at DMLT meeting on the process for defining requirements
    • Gregory Dubois-Felsmann: the question of whether the "all-powerful Butler" model for data access works for all parts of AP, including the ones Simon Krughoff mentioned, is likely to come up soon in the SuperTask/Butler working group as well.
      • Seems like some specific Architecture Group attention to how MOPS and alert distribution will be structured may be needed.
      • Donald Petravick: NCSA also very interested if the Butler-handles-all-production I/O model has exceptions

PDAC statusGregory Dubois-Felsmann
  • Thanks to everyone for getting PDAC to the point where it was possible to demonstrate at the JTM. Everyone's work is very much appreciated.
  • Will be discussing medium-term plans for PDAC at Monday's DMLT. Not ready to discuss in any detail today. Next week's PDAC meeting should be a good venue.
  • WISE data partially loaded - Object-like and ForcedSource-like data loaded by Igor Gaponenko and various Qserv issues that were exposed have been solved. Waiting for information from IPAC on how to access table column metadata.
  • Chris Suberlak of UW is working on providing feedback on the first version of the system

Action items

  •