[Coin-discuss] Thoughts on software review

[Lou Hafer]

Folks,

	Apologies to you non-academics hoping this might die quietly :-).
But even for the non-academic sorts, many of the issues are relevant to
building a reputation for quality software in COIN. It's just the terminology
that changes.
		
I put out the question of review of open source software to the faculty here,
and mostly got the usual (lack of :-) response. From banging up against this
in the past, I can say that a few of my colleagues are opposed on principle.
Not much to be done there. What follows comes from others who find the idea
interesting, combined with some of my own thoughts and ideas batted around at
the INFORMS meeting.

For purposes of tenure and promotion, the categories here are teaching
effectiveness, scholarly activity, and general contributions to the
university and society (aka `service'). In practice, reviewed publication
count dominates; I suspect this is typical.

One thing is clear from everyone I've talked to: to get any credibility, a
software review process must be seen to be credible by people who are
familiar (and thus comfortable) with the present publication review system.
In general, colleagues not involved with software tend to have the vague
impression that creating and disseminating software is an activity on a par
with setting up a personal web site. And to be fair, that's sometimes
correct.  A fairly standard editorial board/reviewer structure will give some
comfort level to colleagues grappling with the notion of open source software
as publication.

With software, we have the advantage that we can establish test suites in
order to validate correctness and evaluate performance.  The subjective
review would need to examine issues of software quality: quality/novelty of
algorithms, implementation (understandability, ease of modification/
enhancement), utility, ease of installation, portability (platforms,
development environments), documentation, etc.

Some of this will only become apparent through use.  One idea would be to
have `Contributed' and `Reviewed' sections in a repository.  `Contributed'
would be software under review, and also software that seems useful but does
not meet review criteria.  After some suitable period (6 months? 12 months?)
software under review could be considered for movement to the `Reviewed'
section. Some sufficiently long time is likely needed in order to estimate
things like utility and commitment to maintenance.

How do we expire software? We'll need the equivalent of back issues.

I don't think we can rely solely on user feedback for review.  There's too
much danger of spotty and self-selected feedback.

A few additional thoughts, in terms of what sort of contribution one makes
with software. This may be relevant in establishing review standards.

It seems to me that there are aspects of both publication and of service.
The initial creation and dissemination of software is analogous to a
publication.  Some software will break new ground. For others, the
contribution might be more akin to a survey paper.  This might be the first
place one sees a complete set of ideas and algorithms collected to form an
implementation.  The software, and the comments with the software, can serve
a valuable purpose by showing how the algorithms fit together and exposing
the details necessary for efficient implementation.

There's also a service aspect, in the sense that you've saved others the time
and trouble of recreating your implementation in order to try some enhancement
of their own.

Subsequent maintenance and enhancement is a mix of publication and service,
depending on the quality of the changes from one release to the next. Fixing
bugs is service; algorithm enhancements move toward publication. One way of
looking at maintaining a piece of software is that it's a service akin to
serving as editor of a publication (but a lot more `hands on').

Somewhere in here we'll need to grapple with the equivalent of `minimum
publishable unit'.  You can't separate the current version of a piece of code
from its past, in the same way that you can publish a short note with a few
references to previous work. Bug fixes and enhancements don't always come as
a discrete module.

							Lou