Current-Infrastructure Meeting 2017-03-16

• Paul will be taking over running current-infrastructure meetings with the next one, i.e., in two weeks time, 30 Mar 2017 

Gregory Dubois-Felsmann, Unknown User (pdomagala), Hsin-Fang Chiang

• RFC-301 - Getting issue details... STATUS covers the request to load this data.
• Consensus seems to have been reached on the acceptability of incremental additions of data without additional RFCs
• Discussion of the means of notifying NCSA staff that a new upload is complete and ready to be made mutable
• Unknown User (pdomagala): would prefer that the request be transmitted as a JIRA ticket
  
  • Unknown User (pdomagala) 30 Mar 2017 make a specific recommendation for the fields to use in these tickets (i.e., component, label, etc.) to ensure that they are seen by the right person and to facilitate summary queries

Unknown User (jalt)

Hsin-Fang Chiang

• HSC PD1 (February public data release) has been loaded to /datasets/hsc/repo - see DM-9683 - Getting issue details... STATUS , RFC-297 - Getting issue details... STATUS

Hsin-Fang Chiang, John Swinbank

• Lauren MacArthur and Hsin-Fang Chiang are collaborating on preparing to run the current versions of DRP code on HSC PD1.
  • It will be done with ctrl_pool
  • Still finalizing exactly what to run. Brief discussion of what to do about meas_mosaic. John Swinbank: it should not be run at this time. Later in the cycle we are likely to run a variety of tests of jointcal and meas_mosaic, and this will be looked at as part of QA work on the pipelines.
  • Planning to start run from latest weekly build but then hold version constant, if possible, for the whole dataset

• Interested in learning more about the plans for data (file) storage, whether object stores will be used, how metadata queries can be turned into DataIDs usable in the Butler.
  • Relevant to some processing cases that don't obviously fit into the current DM pipeline paradigm: MOPS and alert distribution
  • Donald Petravick: current picture is that there need to be three databases involved in this area, enabling
    
    • Finding data by provenance
    • Finding data by metadata describing the file
    • Mapping data identity to data location (so that the "on-disk" arrangement of data can be changed)
  • Donald Petravick: working with Kian-Tat Lim to develop a process for working out requirements, initially, and then design, for provisioning for storage (as well as for computing); will present an introduction at Monday DMLT
    
    • Donald Petravick20 Mar 2017 Make presentation at DMLT meeting on the process for defining requirements
  • Gregory Dubois-Felsmann: the question of whether the "all-powerful Butler" model for data access works for all parts of AP, including the ones Simon Krughoff mentioned, is likely to come up soon in the SuperTask/Butler working group as well.
    • Seems like some specific Architecture Group attention to how MOPS and alert distribution will be structured may be needed.
      • Kian-Tat Lim and Simon Krughoff to meet soon to kick off discussions 20 Mar 2017 
    • Donald Petravick: NCSA also very interested if the Butler-handles-all-production I/O model has exceptions

• Thanks to everyone for getting PDAC to the point where it was possible to demonstrate at the JTM. Everyone's work is very much appreciated.
• Will be discussing medium-term plans for PDAC at Monday's DMLT. Not ready to discuss in any detail today. Next week's PDAC meeting should be a good venue.
• WISE data partially loaded - Object-like and ForcedSource-like data loaded by Igor Gaponenko and various Qserv issues that were exposed have been solved. Waiting for information from IPAC on how to access table column metadata.
  • Gregory Dubois-Felsmann Make sure that Igor Gaponenko and Unknown User (sgroom) get connected to sort out WISE metadata APIs to get this transferred.
• Chris Suberlak of UW is working on providing feedback on the first version of the system