---

---

Overview of Software Reliability

1.  Introduction

Computers and computer systems have become a significant part of our modern society.  It is virtually impossible to conduct many day-to-day activities without the aid of computer systems controlled by software.  As more reliance is placed on these software systems it is essential that they operate in a reliable manner.  Failure to do so can result in high monetary, property or human loss.

The NASA Software Assurance Standard, NASA-STD-8739.8, defines software reliability as a discipline of software assurance that:

1. Defines the requirements for software controlled system fault/failure detection, isolation, and recovery;
2. Reviews the software development processes and products for software error prevention and/or reduced functionality states; and,
3. Defines the process for measuring and analyzing defects and defines/derives the reliability and maintainability factors.

This section will provide software practitioners with a basic overview of software reliability, as well as tools and resources on software reliability as documented in NASA-STD-8739.8.

2.  Software Reliability Techniques

Reliability techniques can be divided into two categories:  Trending and Predictive

• Trending reliability tracks the failure data produced by the software system to develop a reliability operational profile of the system over a specified time.

• Predictive reliability assigns probabilities to the operational profile of a software system; for example, the system has a 5 percent chance of failure over the next 60 operational hours.

In practice, reliability trending is more appropriate for software, whereas predictive reliability is more suitable for hardware.  Trending reliability can be further classified into four categories:  Error Seeding, Failure Rate, Curve Fitting, and Reliability Growth

• Error Seeding:  Estimates the number of errors in a program by using multistage sampling.  Errors are divided into indigenous and induced (seeded) errors.  The unknown number of indigenous errors is estimated from the number of induced errors and the ratio of errors obtained from debugging data.

• Failure Rate:  Is used to study the program failure rate per fault at the failure intervals.  As the number of remaining faults change, the failure rate of the program changes accordingly.

• Curve Fitting:  Uses statistical regression analysis to study the relationship between software complexity and the number of faults in a program, as well as the number of changes, or failure rate.

• Reliability Growth:  Measures and predicts the improvement of reliability programs through the testing process.  Reliability growth also represents the reliability or failure rate of a system as a function of time or the number of test cases.

3.  Software Reliability Metrics

Listed below are some static code and dynamic metrics which are related to software reliability.  The static code metric is divided into three categories with measurements under each:  Line Count, Complexity and Structure, and Object-Oriented metrics.  The dynamic metric has two major measurements:  Failure Rate Data and Problem Reports.

Static Code Metrics

• Line Count:
  • Lines of Code (LOC)
  • Source Lines of Code (SLOC)

• Complexity and Structure:
  • Cyclomatic Complexity (CC)
  • Number of Modules
  • Number of Go To Statements (GOTO)

• Object-Oriented:
  • Number of Classes
  • Weighted Methods per Class (WMC)
  • Coupling Between Objects (CBO)
  • Response for a Class (RFC)
  • Number of Child Classes (NOC)
  • Depth of Inheritance Tree (DIT)

Dynamic Metrics

• Failure Rate Data
• Problem Reports

For descriptions and recommended thresholds of the metrics see Software Metrics and Reliability (ISSRE 1998 Best Paper) by Linda Rosenberg, Ted Hammer, and Jack Shaw.  IEEE International Symposium on Software Reliability Engineering. 1998.

4.  Basic Mathematical Concepts

Mathematically reliability R(t) is the probability that a system will be successful in the interval from time 0 to time t:

where T is a random variable denoting the time-to-failure or failure time.

Unreliability F(t), a measure of failure, is defined as the probability that the system will fail by time t.

In other words, F(t) is the failure distribution function.  The following relationship applies to reliability in general.  The Reliability R(t), is related to failure probability F(t) by:

---

Datasets and Repositories

1.  NASA IV&V Metrics Data Program

The NASA IV&V Facility provides an excellent free Metrics Data Program (MDP) repository.  The repository not only contains software reliability data but also how the data is related to other software metric and artifacts.  Some of the metrics are:

• McCabe Software Metrics
• Halstead Metrics
• Lines of Code Metrics
• Error metrics derived from the association between errors and functions/modules
• Requirement Metrics

See the NASA IV&V MDP website for a full listing and descriptions of the metrics and datasets.

2.  Software Life Cycle Empirical / Experience Database

The Software Life Cycle Empirical/Experience Database (SLED) is hosted by the Data and Analysis Center for Software (DACS).  The SLED contains software empirical lifecycle data.  The data is organized into five datasets:

• Architecture Research Facility (ARF) Error Data
• DACS Productivity Data
• NASA Ames Error/Fault Data
• NASA Software Engineering Laboratory (SEL) Data
• Software Reliability Data

See the SLED webpage for a full description of the datasets.

---

Models and Tools

1.  Models

More than 50 statistical models are currently being used in practice and listed below are some software reliability models separated by category taken from the book entitled Software Reliability by Hoang Pham. Springer-Verlag. ISBN: 981-3083-84-0.  2000.  See this book for full descriptions of these models.

Error Seeding:

• Mills' error seeding model (Mills, 1970)
• Cai's model (Cai, 1998)
• Hypergeometric distribution model (Tohma et al., 1991)

Failure Rate:

• Jelinski and Moranda (Jelinski, 1972)
• Schick and Wolverton (Schick, 1978)
• Jelinski-Moranda geometric (Moranda, 1979)
• Moranda geometric Poisson (Littlewood, 1979)
• Negative-binomial Poisson
• Modified Schick and Wolverton (Sukert, 1977)
• Goel and Okumoto imperfect debugging (Goel, 1979)

Curve Fitting:

• Nonlinear Regression
• Belady and Lehman (Belady, 1976)
• Miller and Sofer (Miller, 1985)

Reliability Growth:

• Coutinho model (Coutinho, 1973)
• Wall and Ferguson model (Wall, 1977)

Non-Homogeneous Poisson Process:

• Musa exponential (Musa, 1987)
• Goel and Okumoto NHPP (Goel, 1979)
• S-shaped growth (Ohba, 1984; Yamada, 1983, 1984a)
• Hyperexponential growth (Huang, 1984; Ohba, 1984, Yamada, 1984b)
• Discrete reliability growth (Yamada, 1985a)
• Testing-effort dependent reliability growth (Yamada, 1986, 1993)
• Generalized NHHP (Pham, 1997a, 1997b, 1999)

Markov Structure:

• Markov with imperfect debugging (Goel, 1979)
• Littlewood Markov (Littlewood, 1979)
• Software safety (Tokuno, 1997; Yamada, 1998)

2.  Tools

Listed are three software reliability tools including some excerpts from each tool's website, where they can be freely downloaded:

SMERFS (Statistical Modeling and Estimation of Reliability Functions for Software) allows the user to perform a complete software reliability analysis. It allows the user to enter either of the two types of model data, modify that data if necessary including transforming it, doing a preliminary model analysis to help select candidate models that are most appropriate for the entered data set, fitting the appropriate models, and then determining the adequacy of the fit. SMERFS allows the user to perform risk analyses with some of these measures to help determine the optimum release time and/or time for reengineering of the software.

CASRE (Computer Aided Software Reliability Estimation) was developed as a software reliability measurement tool that is easier for non-specialists in Software Reliability Engineering to use than many other currently available tools. CASRE incorporates the mathematical modeling capabilities of the public domain tool SMERFS (Statistical Modeling and Estimation of Reliability Functions for Software), and executes using the Microsoft Windows environment.

SARA (Software Assurance Reliability Automation, also known as Non-Parametric Software Reliability) is a comprehensive system which incorporates both reliability growth modeling and design code metrics for analyzing software time between failure data. SARA 1.0 was a research initiative funded by the NASA IV&V Facility via the Software Assurance Technology Center (SATC) located at the Goddard Space Flight Center (GSFC).

---

Additional Resources

1.  Training

Listed are some organizations providing software reliability training:

• RAC
• ReliaSoft
• SoftRel
• SoHaR

2.  Related Sites

Listed are related sites on software reliability, assurance, engineering, and research:

• Bell Canada Software Reliability Laboratory
• Center for Experimental Software Engineering
• Center for Software Engineering
• Centre for Software Reliability
• Data and Analysis Center for Software
• IBM:  Center for Software Engineering
• Software Assurance Technology Center
• Software Engineering Institute
• Software Engineering Laboratory
• Software Technology Support Center