Metamorphic testing

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

Metamorphic testing (MT) is a property-based software testing technique, which can be an effective approach for addressing the test oracle problem and test case generation problem. The test oracle problem is the difficulty of determining the expected outcomes of selected test cases or to determine whether the actual outputs agree with the expected outcomes.

Metamorphic relations (MRs) are necessary properties of the intended functionality of the software, and must involve multiple executions of the software. Consider, for example, a program that implements sin x correct to 100 significant figures; a metamorphic relation for sine functions is "sin (π − x) = sin x". Thus, even though the expected value of sin x1 for the source test case x1 = 1.234 correct to the required accuracy is not known, a follow-up test case x2 = π − 1.234 can be constructed. We can verify whether the actual outputs produced by the program under test from the source test case and the follow-up test case are consistent with the MR in question. Any inconsistency (after taking rounding errors into consideration) indicates a failure of the implementation.

MRs are not limited to programs with numerical inputs or equality relations. As an example, when testing a booking website, a web search for accommodation in Sydney, Australia, returns 1,671 results; are the results of this search correct and complete? This is a test oracle problem. A metamorphic relation could be applied to filter the search by price range or star rating and searching again; the same query term should return a subset of the previous results and a violation of this expectation would be suspicious and reveal a possible bug in the system.

Metamorphic testing was first introduced in a technical report [1] in 1998.

The first major survey of the field of MT was conducted in 2016 [2], followed by another major survey in 2018, which highlights the challenges and opportunities and clarifies common misunderstandings [3]. Although MT was initially proposed as a software verification technique, it was later developed into a paradigm that covers verification, validation, and other types of software quality assessment [4]. MT can be applied independently, and can also be combined with other static and dynamic software analysis techniques such as proving and debugging [5].

In August 2018, Google acquired GraphicsFuzz, a startup from Imperial College London, to apply metamorphic testing to graphics device drivers for Android smartphones[6].


References[edit]

  1. ^ T.Y. Chen, S.C. Cheung, and S.M. Yiu, "Metamorphic testing: a new approach for generating next test cases", Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong (1998).
  2. ^ S. Segura, G. Fraser, A.B. Sanchez, and A. Ruiz-Cortes, "A survey on metamorphic testing", IEEE Transactions on Software Engineering, vol. 42, no. 9, pp. 805-824, 2016.
  3. ^ T.Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T.H. Tse, and Z.Q. Zhou, "Metamorphic testing: A review of challenges and opportunities", ACM Computing Surveys 51 (1): 4:1-4:27 (2018).
  4. ^ Z.Q. Zhou, S. Xiang, and T.Y. Chen, "Metamorphic testing for software quality assessment: A study of search engines", IEEE Transactions on Software Engineering, vol. 42, no. 3, pp. 264-284, 2016.
  5. ^ T.Y. Chen, T.H. Tse, and Z.Q. Zhou, "Semi-proving: an integrated method for program proving, testing, and debugging", IEEE Transactions on Software Engineering, 37 (1): 109-125 (2011).
  6. ^ "GraphicsFuzz is acquired by Google". GraphicsFuzz. 2018-08-06. Retrieved 2018-08-30. 

External links[edit]