Package Maintenance Support
A system for identifying and maintaining package support
Nick West
Last modified: Mon Jun 9 15:18:37 BST 2008
The MINOS experiment is well into exploitation phase and has
accumulated a large body of offline code on which that exploitation
depends. As of June 2008 this is:
- ~150 packages.
- ~1.2M loc - sum .c,.cxx, .h all lines.
Many of the packages are no longer being actively developed and in
some cases the original developers have moved onto other work or have
left the experiment entirely. This leaves us with a support problem:
what happens when development results in a bug that may have been
dormant in an unsupported package or at very least requires diagnostic
work in that package to resolve?
This can be divided into two:-
- How do we assign support to all of our packages?
- How do we maintain this assignment as people come and go?
In order to address both problems a small python based system: PMS
(Package Maintenance Support) has been developed. How it solves
each of these questions is described below.
For every package we identify:-
-
Individual Maintainer
Each package has one or two named individuals who are primarily
responsible for the package. It's possible that a package really is
supported by a group in which case this can just be a lead name who
can be the initial contact for any problem. This assignment may
change with time.
-
Stakeholder Group
Each package is assigned to a group who either played a major part in
its original development or has a particular requirement for its
continued operation. The primary function of the group is continuity.
If an individual seeks to give up responsibility for a package, the
group has to assign a replacement. From this it follows that groups
must have sufficient members that the composition changes slowly with
time and we never have a situation where a group disappears in a
matter of months. This assignment is expected to be stable even though
its composition will change.
For some packages we have very clear individual maintainers and
stakeholder group, for the remainder PMS will perform an analysis of
all the package commit logs to see who has worked on it recently, to
pick candidate individual maintainers, and throughout its development,
to suggest a stakeholder group.
This analysis is carried out at regular intervals. The latest results can be found at:-
http://www-pnp.physics.ox.ac.uk/~west/minos/discussions/pms/
The initial identification of maintainers came, in part, from this
analysis and in part from negotiation with the individuals involved.
Identifying maintainers is only half the battle, this exercise was
undertaken several years ago and the list that it produced slowly fell
out of date, so keeping the list up to date is the other half. PMS
provides some low maintenance method for tracking changes and
spotting holes in support as soon as they appear. It's based on a
system where maintainers have to reaffirm at intervals, their
willingness to continue in the role.
PMS records information about packages and maintainers in a couple of
standard python configuration files:-
- packages.pms
This lists for each package, the type of package it is, the
stakeholder group it is assigned to and a brief description. For
example:-
[PackageMaintenanceSupport]
type=framework
group=core
description=A system for identifying and maintaining package support.
- maintenance.pms
This list has an entry for every person who either maintains a package
or is a member of a stakeholder group. For example
[Nick West]
email=n.west1@physics.ox.ac.uk
cvs_username=west
groups=core
packages=DatabaseInterface,DatabaseMaintenance,Lattice,Navigation,User_Manual
last_confirmation=2008-05-13
The 'cvs_username' is used to map commit logs to individuals. It
could be a comma separated list if the individual uses more than one
username.
The 'groups' entry is a comma separated list of stakeholder groups the individual
belongs to. Group composition is determined by these entries.
The 'packages' entry lists the packages for whom the user is an
individual maintainer. The separate packages file lists group
maintainers.
The 'last_confirmation' records the date the individual last updated
the file.
This information is used by PMS to generate
package_maintenance.html
so that everyone can see who maintains what.
The information can also be used to generate emails to each maintainers
listing the groups they belong to and the packages they individually
support. It is the last entry 'last_confirmation' which is used to
pro-actively track changes.
The system works as follows and involves the PMS utility running as a
daily cron at FNAL.
- It checks individual 'last_confirmation' and for any that are
over some threshold, say 2 months, starts sending emails, every few
days to the individual instructing them to check out maintainers.pms
and update their entry. At the very least they have to change the
'last_confirmation' to bring it up to date. Should they try to
forward date to get a quiet life that too will send email! They can
choose to resign support entirely by removing their entry.
This does require world write access and we have to trust people not
to cheat, but this is CVS so we can track who does what!
- It checks for packages lacking individual or group
maintenance. It also checks for individual entries that have not been
updated in a while. It emails to the core group a list of all such
packages and individuals.
- It generates a fresh copy of the package maintenance list.
In summary:-
- Individuals have control over the packages they elect to maintain
and the groups to which they belong but are required to confirm the
information from time to time.
- The core group quickly learn of real support holes
i.e. orphaned packages, or potential ones i.e. packages whose individual
maintainers no longer acknowledge their responsibility and can then
approach the package's group maintenance to reassign individual
support.