Skip to end of metadata
Go to start of metadata

Date

September 2, 2014 12:00-2:00 PDT

< previous (2014-08-19) | next (2014-09-16) >

Attendees

 

 

Discussion Items

ItemWhoNotes
DiscussionKTL/Jim

Winter2015 Package Reorganization Planning

  • More than a couple of packages may always need a tool
  • Build times affected by SWIG which depends on namespaces and possibly .i file structure
    • AndyS also has a scanner fix to avoid unnecessary rebuilds
  • May need complex layering of geometry to separate Eigen dependency from Qserv
  • JBecla: Qserv has separate build system because it has many modules/products within single repo and package
    • New organization would have multiple Python modules but still have eups package/product and git repo aligned
  • MJ: Why not 1000 packages (and repos)?
    • FE: Disorienting, hard to keep in your head
    • RHL: BaBar had many (GPDF not available)
    • FE: Could do if have conceptual grouping (but tools don't really support)
    • JBosch: Need to build and support our own tooling
      • MJ: Android has something like this
      • JBosch: But may be difficult to have that interact with eups
      • MJ: And still need tooling anyway for more than a couple of packages
  • FE: Doesn't have to be uniform; can split out some small bits for sharing
  • FE: How many repos do I need to touch to do a small task?  Should be small (ideally one)
  • FE: Short build times needed to get fast test turnaround
    • scons should take care of that with multiple modules per package
  • RHL: Don't want people to have to build from source
  • DP: Controlling change, especially deploying to production, difficult with bigger packages; hard to get just changes you want and not ones you didn't
  • RHL: Things that are changing should be minimal in size
    • Others: Not clear that size matters; this has to occur with small or big packages
  • RHL: Working on shared machines is different than independent machines; shared stack helps but requires smaller packages
    • RHL: Want most things to be stable
  • RHL's shared development model:
    • Stack that everyone shares, kept up to date
    • Only packages needed to be built from source are the ones you are working on
    • Essentially treat many packages like internal third-party dependencies
    • KTL: Could use containers to deliver guaranteed environment
    • JBosch: Valuable when working on large volumes of real data; less so for small changes
  • FE: Always need to transition from rapid change to stable deployment; requires release engineer
  • MJ: Worried about pulling in unnecessary things (especially third-party dependencies)
    • Have some of both right now: afw is merged; pex_* and ctrl_* and meas_* are broken up.
    • JBosch: Almost lowest-level code has almost all third-party dependencies
  • JBosch: Could split primitives into stable and unstable
  • Containers: not for everyone, but does solve some problems for people needing pushbutton environments
  • FE: Need a minimum size for reuse to develop a community
  • Splitting out third-party dependencies: cfitsio (would require a lot of refactoring work), minuit, gsl might be possible
    • JBosch: Likely lose more than we gain trying to split these out
  • MJ: We should be like a Linux distribution with a core and lots of optional additions rather than like Fermi/GLAST with everything included
    • FE: This is a spectrum
  • All: Add comments to Jim's page with specific alternate packagings or any desired subset
  • Jim Bosch to add third-party dependencies so that we can see what is/would be brought in by what