Currently we have several scattered git repos relating to MAF.

  • lsst / sims_maf  https://github.com/lsst/sims_maf  -  this contains the core MAF code, including metrics, slicers, and stackers that we'd expect everyone to be able to access (and that have been 'blessed' and code reviewed by us)
  • lsst-nonproject / sims_maf_contrib https://github.com/LSST-nonproject/sims_maf_contrib - this contains community contributed metrics, slicers and stackers, and a few ipython notebooks which use those metrics (mostly as documentation for the metric)
  • lsst-sims / sims_maf_notebook https://github.com/lsst-sims/sims_maf_notebooks - this contains notebooks which demo MAF itself, although it does include a few notebooks which demonstrate particular metrics (not community-contributed metrics)
  • lsst-nonproject / UK Cadence Workshop https://github.com/LSST-nonproject/UK_cadence_workshop_2015  - this contains presentations (PDF and jpg images) from the UK cadence workshop, a few ipython notebooks (including one that is more properly associated with catSim, not MAF), and photos of the pseudo-code written on the whiteboard by the transients group.
  • connolly / ContributedMetrics https://github.com/connolly/ContributedMetrics - this is actually empty except for the README and should go away to avoid confusion! 

 

I suggest to update this as follows:

  • the sims_maf 'core' code be kept as is and should be thought of as the destination for MAF code that is code-reviewed and viewed as an official resource for all community use. Any dependencies for code that becomes part of this repo should be available in the LSST stack. 
    • Add examples/tutorials - this would be a set of ipython notebooks demonstrating the use of THIS VERSION of maf (i.e. when we make changes to the code/API the relevant tutorials here must be updated)
  • the sims_maf_contrib is kept as the official community destination for source that is not necessarily code-reviewed and/or includes dependencies that we will not support through LSST (although any dependencies would ideally be easily installable via pip install). 
    • Reorganize into three directories: 
      • workshops – which will contain directories for each workshop we put on, each of which will contain presentations, pdfs, etc. from that workshop, as well as a page which summarizes metrics & links to the relevant notebooks produced by each workshop
      • notebooks - which will contain directories for several categories of science ('transient', 'static', 'solarsystem', 'variables') and each will contain the relevant notebooks for that area
      • code - which will contain metrics & stacker classes (or other python code) either contributed directly or scraped from the notebooks
  • tidying up old repos: 
    • the sims_maf_notebook repo would go away (and be split among the sims_maf_contrib repo, for notebooks which just demo a particular contributed metric, and the sims_maf repo for documentation about maf, which is the bulk of it)
    • the UK cadence workshop - presentations and jpegs will go into a directory under sims_maf_contrib
    • connolly/ContributedMetrics - should be removed

 

 

 

  • No labels

11 Comments

  1. Simon Krughoff

    Why not move the UK cadence workshop stuff into a subdirectory in sims_maf_contrib?  I liked the idea from the meeting last week where the results of a workshop/meeting could go in a repo and then a parent notebook could hold a table of contents to each of the meetings and the results therein.  Seems like sims_maf_contrib would be the place if sims_mat_notebook is going away.

     

  2. Peter Yoachim

    The only potential downside to the general consolidation is that then we have notebooks in two repos, so maybe that's a minor hurdle for people who want to edit and play with them.

    Should we just commit and say we're going to move to documenting MAF with ipython notebooks?  Seems like a reasonable enough idea, and then we wipe out most of the stuff on confluence and have it point to the repo notebooks.

    This would also be a good time to bump everything up to Jupyter notebooks, especially if the stack python is updating soon.

  3. Andrew Connolly

    How about sims_maf is the core code as stated

    sims_maf_contrib contains the contributed notebooks (which are scraped to put metrics into sims_maf as we need)

    in sims_maf_contrib we organize by "topic" (static, transient, etc) so that it is easy for a new person to come in and find what they need

    under sims_maf_contrib we have a directory for each workshop (which will include material for the workshops) and this will contain a notebook that points to the ones developed at that meeting (and we can note they are obsolete as needed). We also store the presentations under this directory (using github LFS so we don't have to worry about the file size)

    I don't think we can get rid of confluence but we can slim it down (confluence is an ugly interface anyway)

  4. Lynne Jones

    I'm happy with migrating (or at least starting to migrate) our documentation to ipython notebooks instead of confluence. That was why I thought it would be nice to have the notebooks on MAF itself as part of the codebase (to make sure that the versioning stays in sync), but if the consensus is to put it into sims_maf_contrib instead that's fine too. 

    I don't want to create a lot of extra overhead. I don't know if I want to make sure that sims_maf_contrib stays in sync with the code in sims_maf (if we make future big updates), and certainly don't want to be tied to updating it on the same schedule as sims_maf (if, for example, we have a workshop coming up (smile)). But maybe that's still okay, we might just have to add versioning/releases to sims_maf_contrib in that case and just tag it at the same time as we tag a sims_maf release. It just seems simpler to have the 'official' ipython notebook documentation on maf in the same repo as maf itself, so that versioning is easier.

    1. Andrew Connolly

      I like the idea of tagging the notebooks with the maf version they were built for (shame we can't do a setup in the notebook with the version tag). I think keeping the notebooks separate means we don't have to keep them in sync with maf.

      1. Lynne Jones

        I think my point was, if we're using the notebooks as documentation for the use of MAF itself, they should be kept in sync with the maf source, along the lines of "keep the documentation with the code".  

        I don't think documentation of user-contributed metrics needs to keep in sync with the most current MAF version though (along the lines of - there are a few ipython notebooks in sims_maf_contrib that won't run with the current version of MAF, but they still do the job of explaining what is happening in their metrics). 

  5. Lynne Jones

    Andy's suggestion on the sims_maf_contrib subdirectories seems good, although I would like to have all the ipython notebooks about contributed metrics organized by topic instead of by workshop (but we could add workshop directories with links to the relevant notebooks, in their 'topic' directories). 

  6. Andrew Connolly

    I was thinking that the notebooks are organized by topic and then there is a separate directory for workshop materials

     

    workshop --> UK 2015

                     -->allhands 2015

     

    notebooks --> static

                     --> transient

     

    something like that

  7. Andrew Connolly

    Looks good - two questions for sims_maf_contrib

    • code - which will contain metrics & stacker classes (or other python code) either contributed directly or scraped from the notebooks

    If we move code from the notebooks should it go in sims_maf_contrib or sims_maf. Probably a mix as the code in sims_maf will be run by default on all new opsim runs but we will limit the number of these metrics to ones we have vetted. Is that your assessment? We will have to keep an eye on the overlap between the code in each repo

    • notebooks - which will contain directories for several categories of science ('transient', 'static', 'solarsystem', 'variables') and each will contain the relevant notebooks for that area

    If I am eups installing sims_maf will I know where to find the example notebooks (as the code is a little hidden under the layers of LSST code). Would it be worth duplicating the tutorial/example notebooks from sims_maf in sims_maf_contrib

     

     

    1. Lynne Jones

      I don't know if I'd like to duplicate it in sims_maf_contrib, although I'd be willing to move it there instead. 

      The one problem then is that it introduces maf version dependencies in sims_maf_contrib (although maybe that's okay and we don't care, as we'd just commit to keeping sims_maf_contrib up to date with maf, and then add version tags to sims_maf_contrib which match sims_maf). 

      Preference?

  8. Andrew Connolly

    I think moving it makes most sense it terms of getting people to see them. It would just mean keeping the tutorials synced with sims_maf which is what we would do anyway so I agree it is probably not a big deal

     

    other than that I'm happy with it