Date

Attendees

Local: Kenny Lo, Brian Van Klaveren, Unknown User (npease), Igor Gaponenko, Andy Salnikov, Vaikunth Thukral, Andy Hanushevsky

Remote: John Gates

Discussion items

General:

  • Team has started attending CS245 (MW 1:50pm to 2:30pm). Although Vaikunth emailed prof Nate spoke with him after class and he was ok with our presence
    • Slides are usually put up after class, prof recommends reading DB book for lectures (Fritz has it currently)
  • RFC 107: Team consensus is that since we don't have a lot of python code this change will be ok especially if there is a tool that will wrap the docstrings in older code
    • Some concern over who will actually do the work for older code
  • RFC 245: Right now we use debug mostly everywhere, what to do for production? ok to do WARN for now, INFO ok for production?
    • Other comments already on RFC reason well with our requirements
    • AndyS may have some thoughts he will leave on the RFC
    • Need to internally decide which messages need to go on which level
    • The RFC issue may not be sufficiently well defined yet to answer these questions
  • LSST cybersecurity course expired over the break for Vaikunth and Igor, AndyS didn't get it at all - email NCSA?


John:

  • DM-8226: Still having docker pull issues, Vaikunth will help debug it, need to test this on NCSA to be able to merge

  • DM-8426: ORDER BY needs to match output column in the result table, will work on that, major thoughts listed out on JIRA ticket


Kenny:

  • DM-8286, DM-8234: making branches for both and starting work on it, still some machine/helpdesk issues

    • Planning meetings with John etc about these Imgserv tickets

  • Need Parallels license (and VMWare license for John)
  • Thinking about butler, have high level thoughts for future in agreement with current understanding


Brian:

  • DM-8665: Can now extract column/tables with queries. Still some bugs in parser, can't parse some types of queries
    • Want to fix this before finishing or probably make "if clause" for "can parse OK else NOT", can probably handle most SUIT stuff

    • Proof of concept is sufficiently complete


Nate:

  • DM-8125: Working on it and trying to maintain backward compatibility, not sure if it is possible and probably needs further discussion


Igor:

  • Finishing up on LSST20 dataset for KPMs - 82 partitions around the equator have discrepancies but can probably drop this issue since it no longer affects new data

  • After loading is finished will test a bit then replicate on John's side of the cluster. Vaikunth is certain he won't restart the container in the middle of loading now

  • Secondary index might take a while to populate on this data (~4billion rows and InnoDB)

  • Finishing up the documentation for DM-8406

  • Gregory wants us to generate even bigger dataset from WISE (DM-8271 in next sprint)


AndyS:

  • Supertask is on the backburner for now

  • Running L1DB-proto tests on the super machine at in2p3 with different configs, takes time because tests need large amount of data

  • Baseline config is likely not enough for large scale operation, considering splitting it into different aspects


AndyH:

  • Transitioning to xrdssi v2, documentation is a pain

  • AndyH wants to limit size of a request (set to unlimited right now), wants to discuss again with team

  • Cannot optimize if request size isn't limited, MIN should be 2GB can probably do MAX up to 16GB

  • Currently queries cannot be repeated after 16giga queries due to counter, will wrap beyond that

    • Need it for new czar to work with old worker xrdssi - can have switch for old and new type during transition

  • Limited availability in Jan - Gone to CERN next week for 1 week, then San Diego after that, then jury duty last week of Jan


Vaikunth:

  • General issues at in2p3 like QTP index not working anymore

  • Locked memory also coming from mmfsd(gpfsd, ~2GB) and collectd (~32KB) itself, hence non-zero when not running qserv locked tables

    • Can we take away the gpfsd locked memory?
    • Does it work with the containers on all the machines. is the 2GB recorded across all containers or across the machine
  • KPM measurements will be looked at after LSST20 dataset is done loading
  • Updates from Jeff Tseng @ Oxford:
    • Making progress on Edinburgh people, collecting info from colleagues about use cases
    • Checking SDSS help pages, old examples of science queries

    • Talking to people to get more manpower for this work

    • Pan-STARRS made data public - Brian says DESC is aware of this so nothing to do about that yet