SOFTWARE QUALITY ASSURANCE

A.   Concepts and Definitions

Software Quality Assurance (SQA) is defined as a planned and systematic approach to the evaluation of the quality of and adherence to software product standards, processes, and procedures. SQA includes the process of assuring that standards and procedures are established and are followed throughout the software acquisition life cycle. Compliance with agreed-upon standards and procedures is evaluated through process monitoring, product evaluation, and audits. Software development and control processes should include quality assurance approval points, where an SQA evaluation of the product may be done in relation to the applicable standards.

B.   Standards and Procedures

Establishing standards and procedures for software development is critical, since these provide the framework from which the software evolves. Standards are the established criteria to which the software products are compared. Procedures are the established criteria to which the development and control processes are compared. Standards and procedures establish the prescribed methods for developing software; the SQA role is to ensure their existence and adequacy. Proper documentation of standards and procedures is necessary since the SQA activities of process monitoring, product evaluation and auditing rely upon unequivocal definitions to measure project compliance. Types of standards include:

Documentation Standards specify form and content for planning, control, and product documentation and provide consistency throughout a project. The NASA Data Item Descriptions (DIDs) are documentation standards (see Appendix B).

Design Standards specify the form and content of the design product. They provide rules and methods for translating the software requirements into the software design and for representing it in the design documentation.

Code Standards specify the language in which the code is to be written and define any restrictions on use of language features. They define legal language structures, style conventions, rules for data structures and interfaces, and internal code documentation. Procedures are explicit steps to be followed in carrying out a process. All processes should have documented procedures. Examples of processes for which procedures are needed are configuration management, nonconformance reporting and corrective action, testing, and formal inspections.

If developed according to the NASA DID, the Management Plan describes the software development control processes, such as configuration management, for which there have to be procedures, and contains a list of the product standards. Standards are to be documented according to the Standards and Guidelines DID in the Product Specification. The planning activities required to assure that both products and processes comply with designated standards and procedures are described in the QA portion of the Management Plan.

C.   Software Quality Assurance Activities

Product evaluation and process monitoring are the SQA activities that assure the software development and control processes described in the project's Management Plan are correctly carried out and that the project's procedures and standards are followed. Products are monitored for conformance to standards and processes are monitored for conformance to procedures. Audits are a key technique used to perform product evaluation and process monitoring. Review of the Management Plan should ensure that appropriate SQA approval points are built into these processes.

Product evaluation is an SQA activity that assures standards are being followed. Ideally, the first products monitored by SQA should be the project's standards and procedures. SQA assures that clear and achievable standards exist and then evaluates compliance of the software product to the established standards. Product evaluation assures that the software product reflects the requirements of the applicable standard(s) as identified in the Management Plan.

Process monitoring is an SQA activity that ensures that appropriate steps to carry out the process are being followed. SQA monitors processes by comparing the actual steps carried out with those in the documented procedures. The Assurance section of the Management Plan specifies the methods to be used by the SQA process monitoring activity.

A fundamental SQA technique is the audit, which looks at a process and/or a product in depth, comparing them to established procedures and standards. Audits are used to review management, technical, and assurance processes to provide an indication of the quality and status of the software product.

The purpose of an SQA audit is to assure that proper control procedures are being followed, that required documentation is maintained, and that the developer's status reports accurately reflect the status of the activity. The SQA product is an audit report to management consisting of findings and recommendations to bring the development into conformance with standards and/or procedures.

D.   SQA Relationships to Other Assurance Activities

Some of the more important relationships of SQA to other management and assurance activities are described below.

1. Configuration Management Monitoring

SQA assures that software Configuration Management (CM) activities are performed in accordance with the CM plans, standards, and procedures. SQA reviews the CM plans for compliance with software CM policies and requirements and provides follow-up for nonconformances. SQA audits the CM functions for adherence to standards and procedures and prepares reports of its findings.

The CM activities monitored and audited by SQA include baseline control, configuration identification, configuration control, configuration status accounting, and configuration authentication. SQA also monitors and audits the software library. SQA assures that: Baselines are established and consistently maintained for use in subsequent baseline development and control.

Software configuration identification is consistent and accurate with respect to the numbering or naming of computer programs, software modules, software units, and associated software documents. Configuration control is maintained such that the software configuration used in critical phases of testing, acceptance, and delivery is compatible with the associated documentation.

Configuration status accounting is performed accurately including the recording and reporting of data reflecting the software’s configuration identification, proposed changes to the configuration identification, and the implementation status of approved changes.

Software configuration authentication is established by a series of configuration reviews and audits that exhibit the performance required by the software requirements specification and the configuration of the software is accurately reflected in the software design documents. Software development libraries provide for proper handling of software code, documentation, media, and related data in their various forms and versions from the time of their initial approval or acceptance until they have been incorporated into the final media.

Approved changes to baselined software are made properly and consistently in all products, and no unauthorized changes are made.

2. Verification and Validation Monitoring

SQA assures Verification and Validation (V&V) activities by monitoring technical reviews, inspections, and walkthroughs. The SQA role in formal testing is described in the next section. The SQA role in reviews, inspections, and walkthroughs is to observe, participate as needed, and verify that they were properly conducted and documented. SQA also ensures that any actions required are assigned, documented, scheduled, and updated.

Formal software reviews should be conducted at the end of each phase of the life cycle to identify problems and determine whether the interim product meets all applicable requirements. Examples of formal reviews are the Preliminary Design Review (PDR), Critical Design Review (CDR), and Test Readiness Review (TRR). A review looks at the overall picture of the product being developed to see if it satisfies its requirements. Reviews are part of the development process, designed to provide a ready/not-ready decision to begin the next phase. In formal reviews, actual work done is compared with established standards. SQA's main objective in reviews is to assure that the Management and Development Plans have been followed, and that the product is ready to proceed with the next phase of development. Although the decision to proceed is a management decision, SQA is responsible for advising management and participating in the decision.

An inspection or walkthrough is a detailed examination of a product on a step-by-step or line-of-code by line-of-code basis to find errors. For inspections and walkthroughs, SQA assures, at a minimum, that the process is properly completed and that needed follow-up is done. The inspection process may be used to measure compliance to standards.

3. Formal Test Monitoring

SQA assures that formal software testing, such as acceptance testing, is done in accordance with plans and procedures. SQA reviews testing documentation for completeness and adherence to standards. The documentation review includes test plans, test specifications, test procedures, and test reports. SQA monitors testing and provides follow-up on nonconformances. By test monitoring, SQA assures software completeness and readiness for delivery.

The objectives of SQA in monitoring formal software testing are to assure that:

The test procedures are testing the software requirements in accordance with test plans.

The test procedures are verifiable.

The correct or "advertised" version of the software is being tested (by SQA monitoring of the CM activity).

The test procedures are followed.

Nonconformances occurring during testing (that is, any incident not expected in the test procedures) are noted and recorded.

Test reports are accurate and complete.

Regression testing is conducted to assure nonconformances have been corrected.

Resolution of all nonconformances takes place prior to delivery.

Software testing verifies that the software meets its requirements. The quality of testing is assured by

verifying that project requirements are satisfied and that the testing process is in accordance with the test plans and procedures.

E.   Software Quality Assurance During the Software

Acquisition Life Cycle

In addition to the general activities described in subsections C and D, there are phase-specific SQA activities that should be conducted during the Software Acquisition Life Cycle. At the conclusion of each phase, SQA concurrence is a key element in the management decision to initiate the following life cycle phase. Suggested activities for each phase are described below.

1. Software Concept and Initiation Phase

SQA should be involved in both writing and reviewing the Management Plan in order to assure that the processes, procedures, and standards identified in the plan are appropriate, clear, specific, and auditable. During this phase, SQA also provides the QA section of the Management Plan.

2. Software Requirements Phase

During the software requirements phase, SQA assures that software requirements are complete, testable, and properly expressed as functional, performance, and interface requirements.

3. Software Architectural

(Preliminary) Design Phase SQA activities during the architectural (preliminary) design phase include:

· Assuring adherence to approved design standards as designated in the Management Plan.

· Assuring all software requirements are allocated to software components.

· Assuring that a testing verification matrix exists and is kept up to date.

· Assuring the Interface Control Documents are in agreement with the standard in form and content.

· Reviewing PDR documentation and assuring that all action items are resolved.

· Assuring the approved design is placed under configuration management.

4. Software Detailed Design Phase

SQA activities during the detailed design phase include:

· Assuring that approved design standards are followed.

· Assuring that allocated modules are included in the detailed design.

· Assuring that results of design inspections are included in the design.

Reviewing CDR documentation and assuring that all action items are resolved.

5. Software Implementation Phase

SQA activities during the implementation phase include the audit of:

· Results of coding and design activities including the schedule contained in the s/w Development Plan.

· Status of all deliverable items.

· Configuration management activities and the software development library.

· Nonconformance reporting and corrective action system.

6. Software Integration and Test Phase

SQA activities during the integration and test phase include:

Assuring readiness for testing of all deliverable items.

· Assuring that all tests are run according to test plans and procedures and that any nonconformances are reported and resolved.

· Assuring that test reports are complete and correct.

· Certifying that testing is complete and software and documentation are ready for delivery.

· Participating in the Test Readiness Review and assuring all action items are completed.

7. Software Acceptance and Delivery Phase

As a minimum, SQA activities during the software acceptance and delivery phase include assuring the performance of a final configuration audit to demonstrate that all deliverable items are ready for delivery.

8. Software Sustaining Engineering and Operations Phase

During this phase, there will be mini-development cycles to enhance or correct the software. During these development cycles, SQA conducts the appropriate phase-specific activities described above.

F.   Techniques and Tools

SQA should evaluate its needs for assurance tools versus those available off-the-shelf for applicability to the specific project, and must develop the others it requires. Useful tools might include audit and inspection checklists and automatic code standards analyzers.