2017-09-25 Lossy Compression WG Meeting notes

Date

25 Sep 2017

Attendees

• Robert Gruendl

• John Parejko

• Unknown User (bemmons)
• Bob Armstrong

Topic

• How to test compression?
  
  • Data sets to compare compression levels.
  • Algorithms to run on those data sets
• Other compression algorithms worth considering?
  
  • gzip
  • fpack
  • LZMA
  • snappy (stable after being used to compress petabytes of data in Google's production environment)
  • Lizard (formerly LZ5) – recent benchmarks: http://blosc.org/
  • bitpacking (see Section 2.1 of Vohl et. al)

Data sets to test

• Twinkles ("known" sources, sources at and below detection limit)
• Crowded fields (lsst-dev:/datasets/hsc/repo, field=DEEPE02, 03, 06, 07)
• high background fields, e.g. galactic cirrus or nebulosity
• Alerts cutouts (behavior on fields with little background)
• Will also try to obtain DES DM reductions of normal, crowded, and nebulosity (may take a reprocessing to obtain uncompressed versions)

Timing

• How do we test timing?
  • How to deal with caching? Things will be cached, the only way to deal with that in tests would be to have enough data to overwhelm the cache.
  • Do we even consider compression as part of the alerts pipeline?

Action items

• All:  Begin to assemble datasets
• Next discussion should center on a testbed along with measurements to be considered
• Robert Gruendl gets to start trolling SRD documents to determine what requirements are applicable.