Skip to end of metadata
Go to start of metadata




  • Continue progress on PDAC service deployment, discuss user feedback

Discussion items

Infrastructure / NCSAUnknown User (jalt)
  • Discussing how we are going to do OS updates in the PDAC cluster
    • How to apply updates without disrupting operations
    • Gregory Dubois-Felsmann says: we would like 24 hours' notice for unscheduled OS changes and reboots
      • For user-facing services we are only promising 15-30
    • We will start observing NCSA's routine patch day on PDAC: 8-10 Central 3rd Thursday
    • Trey Roby: please post a summary to #dm-pdac channel
      • Particularly concerned to know about reboots and about changes in filesystems
      • Unknown User (jalt): we can take "exclude lists" of packages not to be updated
  • Rough prototype of monitoring system running (based on Grafana), writing a document about it

imgserv cutout services
  • Kenny Lo's work reviewed and on Github, containers built, working on deployment today
    • Brian Van Klaveren will be working on spinning up the new Docker images
    • Kenny Lo will be writing a Confluence page about the current and new APIs
      • Will include "I11" and "I12" as well as some other new work done by John Gates last year
    • Kian-Tat Lim: please get the build-and-deploy process documented
    • Fritz Mueller: looking into using Kubernetes for deployment over the next two months, initially testing at IN2P3

metaservBrian Van Klaveren
  • Both old and new versions of metaserv running, but no actual metadata for the PDAC tables.
  • Waiting for IPAC request for "20 favorite columns"

WISE database loading
  • "Object"-like and "ForcedSource"-like table data downloaded and reformatted for MySQL ingest in December, partitioned for Qserv (~150,000 partitions)
    • Partitioning was on columns with the plain names "ra" and "dec" in both tables, and on the obvious foreign-key relationship between the tables. Referential integrity & consistency between the two spot-checked.
    • Overlap and data location will permit spatial joins between the Stripe 82 and WISE Object-like tables out to a 1 arcmin radius
      • Some discussion of whether the "substripe" scheme is optimal
  • Loading "Object"-like table into Qserv - tables loaded, now generating secondary indexes
    • Expecting to be able to test access through Qserv by the end of the day
  • Loading of "ForcedSource"-like tables awaits a restart of Qserv with updated containers (already built)
  • Igor Gaponenko and Gregory Dubois-Felsmann will recover some work from December to document the IPAC metadata service API

other database / DAX / SLAC
  • John Gates is working on improved handling / scheduling of Qserv queries with large result sizes

  • Trey Roby Early next week we will be ready to deploy our time-series viewer, assuming things work out with the cutout service
    • Questions about user table upload (e.g., to support bulk cone-search queries)
    • Fritz Mueller: Yes, there will be DAX support for this
    • For now, bulk queries will have to be done by iterating over single queries; should work at the level of 20 simultaneous queries being launched (c.f. underlying webserv 40-process pool)
    • Note that our new user is already interested in scales more like 10^3 to 10^5 objects in bulk queries.

Action items