ISO/IEC 29119

The ISO / IEC / IEEE 29119 series consists of five standards. Its stated purpose is to define an internationally agreed set of standards to support software testing.

The first three parts were published in 2013, with the fourth and fifth parts expected to follow in 2015 and 2016 respectively:

• ISO/IEC/IEEE 29119-1: Concepts & Definitions, published September 2013
• ISO/IEC/IEEE 29119-2: Test Processes, published September 2013
• ISO/IEC/IEEE 29119-3: Test Documentation, published September 2013
• ISO/IEC/IEEE 29119-4: Test Techniques.published December 2015
• ISO/IEC/IEEE 29119-5: Keyword Driven Testing

Introduction

ISO/IEC/IEEE 29119 software testing standards are a set of internationally defined documents addressing the software testing concepts, processes, techniques, documents, technologies, and terms..

Currently ISO/IEC/IEEE 29119 has five parts. The set of standards use a layered approach to defining software testing, which is common to many ISO standards. This set of standards presents: test definitions and concepts (part 1); test processes (part 2); test documentation (part 3); test techniques (part 4); and keyword-driven testing (part 5).

ISO/IEC/IEEE 29119 is part of a larger body of ISO/IEEE standards made up of 153 standards within ISO SC7, including ISO/IEC 12207 software life cycle processes and ISO/IEC 15288 systems engineering.

Purpose and Development

ISO/IEC/IEEE 29119 software testing standards are intended to be software domain, environment, and organization independent while supporting a variety of software lifecycles and methods. The process approach is fundamentally risk-based testing and can support test planning and strategy development.

Teams implementing the ISO/IEC/IEEE 29119 standards are adopting and following an internationally recognized standard for software testing. The standard is made available throughout the world and thus provides a basis for an international trading language within the scope of ISO and IEEE using organizations.

These standards were developed using the ISO consensus process (ISO consensus reference to below). The consensus process was reached with a variety of countries and professional organizations that are members of SC7/JTC1 and IEEE. Consensus in ISO/IEEE does not mean that all testers in the world would agree with every idea and concept in the standard, but that the organizations and people involved in the standards’ production did reach agreement. Further, the consensus process does not imply that all viable and new concepts to develop or provide testing goods and services are defined within the standard. Finally, future additions to the standard to address other testing concepts and scope are likely, e.g. model-based testing, etc.

Approach and tailoring

ISO/IEC/IEEE 29119 - 1 provides a common set of terms and concepts to support the other parts. The approach basis for ISO/IEC/IEEE 29119-2 is risk-based. In risk-based testing, risks are used to limit and arrange the test effort within project contexts of problem space, cost and schedule. This is done to solve the problem that testing can be viewed as having an infinite problem space (i.e. complete 100% testing is not possible). . ISO/IEC/IEEE 29119-3 provides definition of common test documents, which can be selected via tailoring as needed. ISO/IEC/IEEE 29119-4 provides structural and functional test techniques, which can also be selected as part of a tailoring activity. Finally, ISO/IEC/IEEE 29119-5 provides definition of keyword-driven test approaches and concepts, for use in project needing keywords. By the use of ISO 29119 heuristics, the test problem can be managed and solved within context of a project or organization Use of the standards can be tailored (requirements removed, changed, or added) in support of contracts or projects to fit local context and needs resulting in varying degrees of adoption. For example, users may select only 2 or 3 document outlines from ISO/IEC/IEEE 29119-3. As per common practice, tailoring needs to be agreed to by stakeholders.

Structure and contents of the standards - Overview of each part

The following sections outline the existing parts of standards that are currently published and/or under development.

ISO/IEC/IEEE 29119-1 (Part 1) - Concepts and Definitions

ISO/IEC/IEEE 29119 Software Testing - Concepts and Definitions, facilitates the use of the other parts of the standard by introducing the vocabulary on which the standard is built and provides examples of its application in practice. Part 1 is informative, providing definitions, a description of the concepts of software testing and ways to apply the software testing process defined in the ISO/IEC/IEEE 29119 part 2 standard and guidance for the other ISO/IEC/IEEE 29119 standards.

ISO/IEC/IEEE 29119-2 (Part 2) - Test Processes

ISO/IEC/IEEE 29119 Software Testing - Test Processes, defines a generic process model for software testing that is intended for use by organizations when performing software testing. It comprises test process descriptions that define the software testing processes at the organizational level, test management level and dynamic test levels. . The processes defined in this standard can be used in conjunction with different software development lifecycle model.

ISO/IEC/IEEE 29119-3 (Part 3) - Test Documentation

ISO/IEC/IEEE 29119 Software Testing - Test Documentation, includes templates and examples of test documentation that are produced during the test process. The templates support ISO/IEC/IEEE 29119-2 Test Processes, i.e. by the test process in which they are produced.

ISO/IEC/IEEE 29119-4 (Part 4) - Test Techniques

ISO/IEC/IEEE 29119 Software Testing - Test Techniques, provides standard definitions of software test design techniques (also known as test case design techniques or test methods) that can be used during the test design and implementation process that is defined in ISO/IEC/IEEE 29119-2. Techniques of part 4 are intended to support part 2 or can be used without part 2.

Suggested Test Design Techniques are categorized in Specification-Based Test Design Techniques, Structure-Based Test Design Techniques, and Experience-Based Test Design Techniques.

Specification-Based Test Design Techniques

Specification-Based Test Design Techniques base on the (functional) specification of the system under test. They are also called black-box testing.

Suggested Test Design Techniques in this group are:

• Equivalence Partitioning
• Classification Tree Method
• Boundary Value Analysis
• Syntax Testing
• Combinatorial Test Design Techniques
• Decision Table Testing
• Cause-Effect Graphing
• State Transition Testing
• Scenario Testing
• Random Testing

Structure-Based Test Design Techniques

Structure-Based Test Design Techniques base on the (internal) structure of the system under test. They are also called white-box testing.

Suggested Test Design Techniques in this group are:

• Branch Testing
• Decision Testing
• Branch Condition Testing
• Branch Condition Combination Testing
• Modified Condition Decision Coverage (MCDC) Testing
• Data Flow Testing

Experience-Based Test Design Techniques

Experience-Based Test Design Techniques rely on the experience of the human tester.

Suggested Test Design Techniques in this group are:

• Error Guessing

ISO/IEC/IEEE 29119-5 (Part 5) – Keyword-Driven Testing

This draft standard will cover keyword-driven testing. Keyword-driven testing is an approach to specifying software tests (normally automated) that is already widely used in the software testing industry. This standard is intended for users who want to create keyword-driven test specifications, create corresponding frameworks, or build test automation based on keywords.

History of ISO/IEC/IEEE 29119

The ISO/IEC/IEEE 29119 set of standards are being developed by working group 26 (WG26) of sub-committee 7 (SC7) of Joint Technical Committee 1 (JTC1) of the International Organization for Standardization and the International Electro-technical Commission in cooperation with the Software & Systems Engineering Standards Committee of the IEEE Computer Society. Development of the set of ISO/IEC/IEEE 29119 software testing standards began in May 2007 by WG26. Members of WG26 are experts and testers from within an international community of testers. WG26 members (http://softwaretestingstandard.org/aboutWG26.php) participated and contributed testing concepts during the drafting, review and consensus process. The standard was approved by consensus and with voting of the ISO country national bodies and organizations such as IEEE. ISO/IEC/IEEE 29119 parts 1, 2 and 3 became official International Standards in September 2013, part 4 in December 2015. ISO/IEC/IEEE 29119 part 5 will be published in the near future.

Replacement of existing standards

The ISO/IEC/IEEE 29119 standards will replace a number of existing software testing standards:

• IEEE 829 Test Documentation
• IEEE 1008 Unit Testing
• BS 7925-1 Vocabulary of Terms in Software Testing
• BS 7925-2 Software Component Testing Standard

These standards will be phased out of support over time.

Standards consensus and intended benefits of the standards

ISO/IEC/IEEE 29119 software test standards were created and approved by ISO and IEEE organization following their standard development methods. This means a group of people, countries, and organizations developed the standard. The standard was approved by multiple testers and countries. The development and approval process is by consensus, which means that a majority of the standard development participants (not the whole world of software tests) agrees with the document. IEEE approved ISO/IEC/IEEE 29119 within the standards association group. And for ISO, SC7 member countries approved ISO/IEC/IEEE 29119 by majority. Membership in these are subject to ISO and IEEE rules and is an open process with these organizations, though not every tester in the world can be involved. The standards were not intended to be state of the art or capture all practices in testing. New concepts and different testing practices will be added in future updates or new parts as ISO/IEC/IEEE 29119 evolves. Potential users of the standards may include:

• Companies needing common baseline of testing concepts, terms, and ideas (an organization to organization trading language to help promote national and international business);
• Government organizations and regulators needing an ISO/IEC/ISO industry approved baseline;
• Test organization needing an industry based starting point;
• Purchasers and procurers of software products to assess third parties;
• Developers of software, e.g. to support unit level testing;
• Regulatory authorities wishing to provide accreditation services against the standards;
• Professional training course developers and providers;
• University lecturers wishing to teach their students content from the standards;
• University researchers wishing to conduct research and collect data based on the standards;
• Software testing tool vendors wishing to implement concepts from the standards;
• Organizations wishing to gain accreditation to the standard.

Future plans for maintenance of standards

All ISO standards are reviewed, revised, and if necessary retired no later than 5 years after publishing. ISO/IEC/IEEE 29119 naturally follows the same process. Interested parties can provide input to the standard for the systematic review, either via the ISO national standard bodies, or by directly approaching the standard organizations including ISO, IEC or IEEE standards association. The decision about a revision of the standards will be based on the available feedback, comments, input, consensus and voting. The future plans for ISO/IEC/IEEE 29119 parts will attempt to address as many current issues and controversy items as consensus will allow. The impact and proof by use of ISO/IEC/IEEE 29119 parts remain future work. Before the viability of the standard can be determined, the standard had to be developed and made available for use, which has happened. The development represented what participants understood to be common practice in their experience in countries, companies, and organization. These groups will be the initial users of the standard and will provide evidence of success or failure.

Controversy

There is resistance from sections of the testing community towards the standardization of software testing. In particular, the context-driven community and the International Society for Software Testing reject these standards. In short, the reason for opposition is that ISO-29119 effectively excludes Context Driven Testing and at the same time Context Driven Testing won’t agree to any simplistic standard based on templates or simple formulae.

External links

• International Organization for Standardization
• ISO/IEC/IEEE 29119 Software Testing
• International Society for Software Testing
• An open letter to the President of the International Organization for Standardization about ISO 29119
• "IEC 29119-1", "IEC 29119-2", "IEC 29119-3", "IEC 29119-4", "IEC 29119-5" at International Electrotechnical Commission