Bayesian nets provide radical improvements in software
defect prediction
January 2007
The developers of any new complex software system will
confirm that, no matter how much testing they perform, there will still be
plenty of defects (or ‘bugs’) yet to be found. The hope is that, when the
software is released, any defects found by end-users will have minimal impact.
Hence, the decision about when to stop testing and release the software must
always be balanced by the likely number (and criticality) of remaining defects.
It follows that the ability to produce accurate predictions of ‘residual’
defects in software systems is one of the most important and challenging tasks
confronting software engineers. It is especially relevant for safety critical
software (such as in transport and medical systems where software that is
released with too many defects can have life-threatening impact); but the
business of any commercial software producer can be devastated if they get
their release decision wrong.
In 1999 Prof Norman Fenton and Dr Martin Neil wrote a paper
[1] about software defect prediction that was published in the IEEE
Transactions on Software Engineering. The paper was a critique of the commonly
used approaches to the problem and they proposed a novel approach based on
causal models. Although the 1999 paper provided no validation of the new
approach, it became one of the most cited papers in computer science. According
to Essential Science Indicators the paper is in the top 1% most
influential papers.
In 2006 the editors of the journal Information and
Software Technology invited the authors of the 20 most cited papers in
software engineering since 2000 to submit a new paper for a special edition of
the journal. The intention was that the papers (which would still be subject to
the journal’s rigorous reviewing process) should reflect on the work that was
done and report on new developments. Fenton and Neil were keen to respond to
this invitation since their work on defect prediction had recently taken a new and
exciting turn.
Ever since the 1999 paper Fenton and Neil had been working
with Phillips Consumer Electronics to evolve and validate the causal models.
Phillips saw the work as especially important because they developed complex
software that was embedded in electronic devices like TVs and DVDs. Being able
to improve their decision-making about when to release the software was
critical from a business perspective because faulty software could lead to the
recall of entire batches of TVs or DVDs. Using models that were built and
executed in the AgenaRisk toolkit Phillips were eventually able to perform a
comprehensive validation on 36 major projects based in Bangalore and Eindhoven.
The ‘headline’ result was that Fenton and Neil’s 1999 conjecture that causal
models could outperform traditional approaches was well and truly proven. The
predictive accuracy of the causal models was 93% (comparing predicted with
actual numbers of defects) compared with 80% that could be achieved with the
best of the traditional approaches using the same project data. But even more
important than the predictive accuracy was that the causal models and tools
enabled project managers to do genuine risk assessment and ‘what-if’ analysis
that simply was not possible before.
In addition to the exciting results with Phillips the new
paper also describes how the approach can be easily tailored to handle
arbitrary software development processes. Hence the approach can be used by
organisations whose development processes are very different to those of
Phillips.
The new paper [2] appears in the January 2007 issue of
Information and Software Technology (Volume 49, pages 32-43). A preprint
version can be found here:
http://www.dcs.qmw.ac.uk/~norman/papers/ist_fenton.pdf
A simplified version of the defects prediction model in action can be seen here.
References
[1] Fenton NE and Neil M, ''A Critique of Software Defect
Prediction Models'', 25(5) IEEE Transactions on Software Engineering,
675-689,1999.
[2] Fenton NE, Neil M, Hearty P, Marsh W, Marquez D, Krause P, Mishra
R, "Predicting Software Defects in Varying Development Lifecycles using
Bayesian Nets", Information & Software Technology 2006
