Data Management
Space shortcuts
Page tree
Skip to end of metadata
Go to start of metadata

Date

Attendees

Discussion items

Structure of JSON results

  • Relevant links
  • JSON object with extra info is better than arrays
  • unofficial standard: response as top level key
  • perhaps try to come up with a generic response container object, containing:
    • top level response key
    • pagination information
    • metadata about json object
    • status, eg "no rows returned"
  • server or client error, these are tricky (user provided input bad, server down, etc). Sometime it is a problem with application software. Return 500 or 404 type error. 
  • We should define our own errors that correspond to the special error conditions
  • software for validating json exists, not as much as for xml, it is all much looser 
  • useful for documenting json results:
  • might be useful to have unit tests to check we are returning what we should
  • Will production be using the RESTful API / webservice?
    • webservice will be layered on top of internal services (butler, cutout etc).
    • no compelling argument that production computation should use RESTful interfaces
    • so use the underlying service, don't go through a central webservice
    • need to check if cutout is modeled in storage model as distributed service
      • sending the same image to many nodes == less caching
      • rely on data locality (don't send the same data to many nodes, send to the same)

Fast image search

  • relevant links
    • DM-1150 - Getting issue details... STATUS
  • SciSQL does not support what we need for searching images
  • concerned about indexing (finding fast whether polygons overlap)
  • if images of different size, need to pick some coarse level of htm, that is suboptimal
    • can store multiple htm levels
  • likely need special index, eg 3-d R-tree
  • Talk to mysql GIS experts at: http://www.percona.com/live/mysql-conference-2015/
  • in Fermi, dealing with "events" that are point, we deal with areas 
  • science question: what do we need to support in terms of searching / selecting images. E.g., point in image enough? Polygons overlapping images. Jacek will ask Mario Juric. Via JIRA ticket.
  • Serge will update JIRA ticket: add notes what to test

Raw images

  • Related links
    DM-2241 - Getting issue details... STATUS
  • Format?
    • SUI can always request the image as FITS
  • Returned data?
    • Raw pixel plus metadata we obtained from obs control system and telescope control system (which will include "where telescope was pointing" – approx WCS)
    • few years later, when we have precise WCS, we might want to return that with image. But might want to call it differently ("augmented raw image") 
  • Size units?
    • typically amplifier level
    • ok to request smaller than amplifier (although it is usually not useful)
    • There is no such thing as CCD raw image. No plans to do stitching raw images
  • Typical uses of raw images? What metadata should go with raw data? Xiuqin will ask Mario Juric. Via JIRA ticket

Next week - Xiuqin will run the meeting