Written by Mario Juric, 2012-04-05
Writing good documentation is hard. It takes experience, effort, and some talent. Writing one that is good enough is not. Numerous developer-documented open source projects prove it's possible, and essential.
Our code is unevenly documented. While there are exceptions, most of the documentation is either function-level (autogenerated by doxygen), or scattered on the wiki with little organization or context. Most worryingly, some (critical) packages have little or no documentation at all.
Due to this, we're beginning to observe the following:
The primary argument against documenting is that it distracts from writing code. In a large, distributed, team, where the age of an average developer is ~3 years (of work on the project), not documenting also distracts from writing code. Most insidiously, most do not feel this intuitively: while we all tend to remember the hateful hours spent trying to churn out a paragraph of documentation, we tend to forget the hours spent explaining things in person or via e-mail.
The other argument against documenting is that developers are incapable of producing good documentation, and that it's a technical writers' job to do it. Firstly, developers are capable of producing good enough documentation. However, even if we had a technical writer (which we currently don't), they require a starting point on which to expand on. Technical writers are not a tool to get from no documentation to some documentation, they're a tool to get from 'good enough' documentation to 'great documentation'. Having some developer-written notes is a prerequisite for hiring a technical writer.
As with everything, we need to strike the right balance. Asking for excellent documentation that is up-to-date at all times is obviously unreasonable. But having no documentation at all, or just doxygen-level documentation, is equally undesirable. We need to strive for the middle ground: documentation that is "good enough", sufficient for new developers (as well as the authors themselves!) to understand the code, and sufficient for management to track progress and plan future development.
This document is an attempt to set guidelines for how to write documentation what is 'good enough', but does not overtax our limited resources.
I propose having three levels of documentation:
System Architecture Overview and Developer Guide
A high level overview of the whole DM system. It needs to connect the actual coded packages to high-level designs from LDM-XXX documents. For example, LDM-151 chapter 2.2.3 describes the DRP pipeline. This document needs to show how the current prototype code (pipe_tasks et al.) maps to the designed pipeline (note: it's not obvious to me that these should be separate documents).
A second purpose of this document is to get new developers started. The text that is in devenv/doc seems like a great start. This document should link to package-level documentation for more details. For ease of editing, the master copy of this document should live on the wiki. Help will be provided to keep this document up-to-date (e.g., by Frossie or Mario).
Package-level documentation
This is the entry point for a new developer looking at the package for the first time, and it must quickly answer what the package does, why, and how. It can begin with something as simple as a README file, growing to full-fledged reST, wiki, or other documentation as more details are added. Alternatively, the README can hold pointers to other places where non-autogenerated documentation can be found; if it's on the wiki, the link should be of the form LSST/Documentation/<package_name>, to maintain consistency. Just having a pointer to doxygen output does not qualify as having provided documentation. Note: it is very important that package-level docs are kept up-to-date, and it is the developers' and reviewers' sole responsibility to make it so.
API (reference) documentation
This is documentation auto-generated from the sources (e.g., Doxygen). Every public API function must be documented. Private functions (implementation details) should be documented in the code, but must not be exported to public (doxygen generated) documentation. Doing otherwise clutters the documentation and exposes implementation details to the world.
Package-level documentation needs to provide sufficient detail to allow a reasonably skilled developer to discern what it does, how it does it, and why. That is its purpose, no more, no less.
At minimum, it should provide answers to the following:
What the package does
Overview of what the package does (the intent). This should be no longer than 10-15 sentences, sufficient for the reader to quickly understand what is the purpose of the package.
Who and how should use it
Usage description. This may be written as a short general introduction, plus examples on how to perform the most common user stories defined for the package. If any examples are bundled with the package, it should have pointers to those. Note that all examples have to be documented; having a directory of undocumented .py files with cryptic names does not qualify as having provided example code.
Architecture overview (how it does it)
This should provide a high-level overview of the architecture of the package, sufficient for the reader to learn about the details from reference documentation. It must explain any non-obvious design decisions or unconventional implementation details (can be in form of a F.A.Q.). It should contain a historical introduction, if it helps the reader understand how the package came to be the way it is (e.g., this package started as . . . but was then split off when it was deemed too complex). Finally, it should note any identified deficiencies and TODO items (i.e., expected future development).
Notes
Any other relevant information a developer or user should know about. For example, if there was data included in the package, where did it come from.
Changelog
A brief log of most important changes.
References
Links to autogenerated documentation (doxygen/sphinx), or to related packages.
Maintainer and Authors
Who is the current maintainer of the package. It is also customary to acknowledge the major contributor(s), or initial author(s).
If this documentation resides in a README or other text files within the package, it is recommended reStructuredText (reST) is used as markup. This makes the documentation source easy to read, yet machine readable and usable by tools like Sphinx or similar. LaTeX, in particular, should be avoided; LaTeX sources are notoriously difficult to read.
Some examples can be found at:
Unfortunately, the latter two of these are unusual, as they describe packages that deal either with data or developer support tools. More relevant examples should be added as soon as they appear.
Documentation should be written once a design is stable, and an initial implementation exists. Documents written for design reviews are usually good starting points.
In particular, no tickets should be merged to master without the reviewer having made sure that:
Note that 1) will apply to almost every merge, 2) will be needed for features that add or change major features of the package, and 3) will be required only when changes to the package have a significant impact to the overal design (examples: replacement of old pipeline code with pipe_base/pipe_tasks, or pex_policy with pex_config.
In the words of Django's Jacob Kaplan-Moss:
"Auto-generated documentation is almost worthless. At best it’s a slightly improved version of simply browsing through the source, but most of the time it’s easier just to read the source than to navigate the bullshit that these autodoc tools produce. About the only thing auto-generated documentation is good for is filling printed pages when contracts dictate delivery of a certain number of pages of documentation. I feel a particularly deep form of rage every time I click on a “documentation” link and see auto-generated documentation.
There’s no substitute for documentation written, organized, and edited by hand.
I’ll even go further and say that auto-generated documentation is worse than useless: it lets maintainers fool themselves into thinking they have documentation, thus putting off actually writing good reference by hand. If you don’t have documentation just admit to it. Maybe a volunteer will offer to write some! But don’t lie and give me that auto-documentation crap."
An e-mail to lsst-data is just one notch above (below?) talking to someone in person. It should be considered transient.
See above.