Talk:Regression testing

From Wikipedia, the free encyclopedia
Jump to: navigation, search
WikiProject Computing (Rated Start-class)
WikiProject icon This article is within the scope of WikiProject Computing, a collaborative effort to improve the coverage of computers, computing, and information technology on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.
Start-Class article Start  This article has been rated as Start-Class on the project's quality scale.
 ???  This article has not yet received a rating on the project's importance scale.

Older discussions[edit]

Does anyone know how to document regression testing or any other kind of testion for that matter?Stanko 19:10, 27 September 2006 (UTC)Stanko

I removed this line, as it clearly belongs in talk and not in the article itself: Remaining questions - who invented the term? Who was the first to use it systematically?

And I'm not sure about this quote (I left it in, but added quotations marks to make clear that it's a quote.)

"As a consequence of the introduction of new bugs, program maintenance requires far more system testing per statement written than any other programming. Theoretically, after each fix one must run the entire batch of test cases previously run against the system, to ensure that it has not been damaged in an obscure way. In practice, such regression testing must indeed approximate this theoretical idea, and it is very costly." -- Fred Brooks, Mythical Man Month (p 122)



The article would really need some references/external links. Anyone having some? Cheers - David Björklund 11:21, 13 Mar 2005 (UTC)

Relation with extreme programming[edit]

The last paragraph states that "regression testing is an integral part of the extreme programming software development methodology". This is IMO abusive since it is part of many, if not all, of those methodologies. Furthermore, the article about extreme programming does not mention regression testing, which tends to prove my point. Regression testing is merely a software engineering trick that many of us use (whereas all should). - 15:58, 25 October 2005 (UTC)

I concur. Extreme or Agile programming has a different view with regards to testing in that it preaches to write tests before implementing features, something which is less common and seldom explicitly preached in other development methods. In other methodologies it's not necessarily discouraged or not practiced; it's just not explict as one of the most import principles. Wouter Lievens 08:06, 7 April 2006 (UTC)
I think that eXtreme Programming preaches to write tests before implementing features for a few reasons. One of those reasons is so that you have a full suite of unit tests to run as a regression test at any time. And I think that the reason for that is because in extreme situations, there can be no tolerance for slippage, or regression. Every update to the code must be a step forward.
Like mentioned, regression testing is a trick that many of us use and most of us should. But I think that eXtreme Programming demands it. Maybe it has more to do with Test Driven Development. And maybe someone should update the eXtreme Programming article to mention regression testing. DRogers 12:28, 19 March 2007 (UTC)

Why does this apply only to software?[edit]

I hoped to see a definition of regression testing that applied to systems in general, not just software. Is there some reason this definition shouldn't be applied to entire systems, subsystems, and integrated hardware/software units? —The preceding unsigned comment was added by (talkcontribs) 23:53, 25 September 2006 (UTC).

Well, the best answer I can give is that software people started this article and software people maintain it. Regression testing as applied to software is all I know. If you think that it applies to something else, I encourage you to start an article. DRogers 12:19, 19 March 2007 (UTC)

Regression testing is one of the key elements when designing integrated circuits, ASICs and FPGA based designs. In this field there are quite a lot of specialiced tools on the marked to help designers with it. On this top 10 asic design and verfication list, regression testing is on the second place. the problem is a bit, that this article belongs to the software testing portal (which is ok) and that there is no article about asic design methods and verification where it would also belong to. -- (talk) 07:13, 20 April 2011 (UTC)

It belongs to Wikipedia. If you want to add a section on other areas, feel free to. You could also create a different article. It just sounds like there are different methods involved though and the concept remains the same. --Walter Görlitz (talk) 13:52, 20 April 2011 (UTC)

Important questions[edit]

The article does not answer 2 important questions:

  1. What does the word "regression" in this context mean?
  2. And why are the tests called "regression tests"?

I recommend you to read the much better article "Regression Testing" at by Adam Kolawa

—The preceding unsigned comment was added by (talkcontribs) 09:58, 28 March 2007 (UTC1)

regression as opposed to progression - a step back not forward.
From Latin 'gressus' - Perfect active participle of gradior (“‘step, go, walk’”).
-- (talk) 11:00, 17 September 2009 (UTC)

(NJgeezer (talk) 15:51, 16 November 2010 (UTC)) The Wiley article has this definition: "Regression testing identifies when code modifications cause previously-working functionality to regress, or fail..." The important point here is that the existing def in this wiki article does not explain the added nuance of "regression" , just "testing". The function of regression testing is not to test the change itself (commonly known as unit testing), but to ensure through broader testing that the change has not caused a different bug. This will almost always involve programs or modules outside the scope of the actual code changes. e.g. a change to a DB extract may have resolved a missing country's records, but those "new" records may contain data that causes a completely separate, downstream system to crash since their code does not allow for that new country's currency code. So we are testing not that the change works, but that it has not caused any application regression (loss of functionality) i.e. a failure of previously working, unchanged code.

Types of regression[edit], I removed your edits. First, I don't think that those are types of regression. Maybe they would belong under the risk mitigation section. And in that case, I think "Complete test suite repetition" and "Partial test repetition based on traceability and analysis of technical and business risks" cover your points. Does anyone disagree? DRogers 12:48, 23 May 2007 (UTC)

Added citations and a testing resource[edit]

Since this topic is lacking cited content, I added some from Albert Savoia and Kent Beck. Kent Beck is the father of XP and JUnit (although I deliberately avoided making the new content Java-specific and stuck with what would transcend multiple languages). Alberto Savoia has written extensively on unit testing, regression testing, and developer testing.

In order to avoid tying content to any language, I dropped "JUnit Factory" down to the bottom as an external link instead of making it a reference.

Cheers. MickeyWiki (talk) 16:25, 29 November 2007 (UTC)

first paragraph a bit mistakable?[edit]

I dont know what regression testing is - but as i understand the explanation here, it is a test applied just after changes to software have been made. These changes may be "normal" changes or bugfixes. If this definition is correct, i find the the second sentence "The intend of regression testing is to assure that a bug fix has been successfully corrected.." mistakable; this sounds to me like "it is a test, that is applied after trying to fix a bugfix". So there was a mistake _fixing a bug_? I think this is too special..

In the case that my interpretation is correct, i suggest following two sentences for the beginning:

Regression testing is any type of software testing that seeks to uncover software errors after changes to the program (e.g. bugfixes) have been made, by partially retesting the program. The intent of regression testing is to assure that a bugfix itself is correct, or more general, that a change did not introduce new bugs.

and maybe the next sentence should be shorted as to not become repetitive.

len48 —Preceding unsigned comment added by (talk) 19:07, 30 December 2009 (UTC)

I second the motion but don't feel comfortable making the change myself. If you need more evidence: "Regression testing is performed after making a functional improvement or repair to the program." [The Art of Software Testing, by G. Myers] (talk) 22:15, 22 October 2010 (UTC)
I updated it taking into account what has been noted here. AliveFreeHappy (talk) 23:10, 22 October 2010 (UTC)

Is this claim too bold?[edit]

In the corporate world, regression testing has traditionally been performed by a software quality assurance team after the development team has completed work. However, defects found at this stage are the most costly to fix.

This latter statement in the background section seems to be a bit odd and does not have any citation or reference. I would have thought defects found after QA would be even more expensive, but does agree that QA tests are more costly then dev team tests. — Preceding unsigned comment added by (talk) 10:20, 10 June 2011 (UTC)

Both are correct. There have been several studies that show increasing cost for bug fixes at each stage of development. It is 100 time more expensive to fix the bug if found by the client than if found at the time of creating requirements. However, the phrase above just needs a little tweak: "defects found at this stage are the more costly to fix than if found earlier." --Walter Görlitz (talk) 13:50, 10 June 2011 (UTC)

Regression Testing vs Non-regression testing[edit]

Should this article be merged with Non-regression_testing? What is the difference between these two terms (I do not see any). Is one of the terms more correct than the other (I was always saying Regression Testing, but recently met people that always say Non-regression Testing)? — Preceding unsigned comment added by Loomchild (talkcontribs) 14:19, 10 January 2012 (UTC)

No. The concepts are different, and "NRT" is unsupported in the literature. --Walter Görlitz (talk) 15:32, 10 January 2012 (UTC)
Thanks for explanation and for raising this topic on NRT Talk page. --Loomchild (talk) 14:27, 11 January 2012 (UTC)

edits by anon Woodbridge, Connecticut[edit]

These edits are not correct. You want to find errors, not defects. The errors are created by developers and they are not defects until someone actually triggers them. Also, defect is a euphemism for a software bug which is described as "A software bug is the common term used to describe an error, flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways." This is why we should not be using defect. As for "or other software properties", what sort of weasel words are those? Functionality is all that is important in regression testing. There are no other relevant "software properties" (whatever that means). --Walter Görlitz (talk) 22:59, 28 February 2012 (UTC)

I have no hope of convincing Walter but here are some definitions that should illuminate the two issues. Emphasis added.

error – “a human ACTION that produces an incorrect result, such as software containing a fault" (ISO 24765:2010, Systems & Software Engineering Vocabulary)

defect – “a problem which, IF NOT CORRECTED, could CAUSE an application to either fail or to produce incorrect results” (Id.)

regression testing – “testing required to determine that a change to a system component has not adversely affected functionality, RELIABILITY OR PERFORMANCE, and has not introduced additional defects” (Id. Also: ISO 90003:2004, Software Engineering Guidelines for the Application of ISO 9001:2000 to Computer Software)

The other option is to argue with international standards. — Preceding unsigned comment added by (talk) 02:44, 29 February 2012 (UTC)

It's not about convincing me, it's about not being vague and not using terms in incorrect ways. --Walter Görlitz (talk) 03:09, 29 February 2012 (UTC)

I gave my sources of authority. What are yours? — Preceding unsigned comment added by (talk) 03:30, 29 February 2012 (UTC)

A library of a dozen books, fifteen years of Better Software Magazine and twenty years in the industry. Many of the testing articles on Wikipedia were either started or heavily edited early on by me.
The fact that you have sources don't mean that your additions are any more clear than what was there. Your addition of "or other software properties" is vague and your sources (which are not listed in the article per WP:V and WP:RS) don't help make it less vague. The fact that your ISO terminology differs from some of the rest of the industry doesn't mean that it's right, its one opinion. Check the software bug article to compare with consensus. --Walter Görlitz (talk) 03:35, 29 February 2012 (UTC)

ISO standards come into existence through world-wide consensus among experts in the subject domain. May I ask one more time for the precise sources for your def’ns of error, defect and regression testing, with quotation of these definitions, so everybody can compare authority and actual words. (talk) 04:02, 29 February 2012 (UTC)

Check the software bug article. --Walter Görlitz (talk) 04:16, 29 February 2012 (UTC)

That's what I did and that's why I made the changes - just in the first sentence, for a start, the article needs it. If the present version is representative, nobody should be surprised by the sorry outcomes of much of the software engineering industry. Cheers, (talk) 04:47, 29 February 2012 (UTC)

The only sorry thing is your addition of ambiguous statements. Please expand the "or other software properties" phrase or remove it. --Walter Görlitz (talk) 04:56, 29 February 2012 (UTC)

Expanded phrase, done. Thanks. (talk) 05:05, 29 February 2012 (UTC)

Now that I have time to read the whole I understand. You're confusing functionality and functional testing, which is why you're comparing it to non-functional requirements. Functionality is not the same. Fixing. --Walter Görlitz (talk) 06:13, 29 February 2012 (UTC)

No, you don't understand the whole - but that's ok. Your edit sounds as if it comes from a German or Dutch speaker. No verbs at the end of a sentence. Try again. (talk) 07:45, 29 February 2012 (UTC)

Fixed the incorrect change to the relative clause. --Walter Görlitz (talk) 14:15, 29 February 2012 (UTC)

Good to see “error” removed and “non-functional” introduced. The non-functional aspects are the “number one risk” in software engineering. (CrossTalk, April2007, p.12) Being such a risk the non-functionals need full attention by regression testing. (talk) 15:55, 29 February 2012 (UTC)

But regression testing doesn't usually include non-functional testing despite it being the number one risk. We're not creating policy but should be reflecting actual use. --Walter Görlitz (talk) 15:57, 29 February 2012 (UTC)

You have to start moving to different circles. (talk) 16:12, 29 February 2012 (UTC)

Vague definition[edit]

A few comments on the heading of this article, if I may:

"Regression testing is a type of software testing that seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes such as enhancements, patches or configuration changes, have been made to them."

Here, "regressions" are equated with "new software bugs". 'Regressing', or 'taking a step back', clearly points to the past, whereas 'new' does not. Regression testing seeks to verify that a system has not regressed to a prior, less functional or less operational state after the introduction of changes.

"The purpose of regression testing is to ensure that changes such as those mentioned above have not introduced new faults."

Regression tests are pre-existing tests (made during earlier development) that either test functionality or bug fixes, and as such can only test for known conditions, not new ones. The kinds of tests suggested here regarding enhancements etc. might be called progression tests.

To illustrate: in the latter case (testing bug fixes), when a bug is found during use of a system, a test case is written that reproduces the occurrence of the bug, testing for the expected outcome. When the bug is fixed, the test will pass. These fixes and tests are committed to the system's source repository to ensure the bug does not re-appear unnoticed. While fixing such a bug, the test is usually classified as a unit test or functional test, depending on the scope. In future software revisions, re-running those earlier tests is what constitutes regression testing.

"One of the main reasons for regression testing is to determine whether a change in one part of the software affects other parts of the software."

This is, IMHO, besides the point; such determinations are the purpose of Integration testing.

"Common methods of regression testing include rerunning previously completed tests and checking whether program behavior has changed and whether previously fixed faults have re-emerged."

This is more to the point; however, regression testing only consists of rerunning previous tests. Any new tests added will not be regression tests until they are re-run in the future; while they are new, they are unit/functional/integration/what-have-you tests. Also, "checking whether program behavior has changed" is too vague of a purpose; perhaps "verifying program behaviour has not changed" is more precise.

"Regression testing can be performed to test a system efficiently by systematically selecting the appropriate minimum set of tests needed to adequately cover a particular change."

I would expect the heading to present a clear, concise definition and not go into details about the possibility of optimizing for efficiency. Further, it suggests that one of the purposes of regression testing is to be able to only select those tests that test the impact of a change, which it is not. Typically, regression tests are "blind": they test system functionality to ensure a change does not have unintended side-effects, which goes counter to being able to adequately determine all the effects of any change.

"Contrast with non-regression testing (usually validation-test for a new issue), which aims to verify whether, after introducing or updating a given software application, the change has had the intended effect."

Now this is good; and it is easily seen that when future changes are made, re-running such a test is considered a regression test as it will assert that the previously added functionality is still in effect.

Kenneyw (talk) 13:17, 27 June 2015 (UTC)

Regression Testing[edit]

Hey, I had edited the content of regression testing in Wikipedia page and was left with your comment

"Revert as WP:OR the content is focused on web testing and was almost entirely unreferenced. The writing style was inappropriate with a lot of bullet lists. Refs do not go into headings"

1) I had the same feeling that there were a lot of bullet points and am working on correcting them.

2) I had posted a few references from published journal papers. Are journal papers not a good source for content?

3) And about the web testing, some of the tools were mentioned and not all. NarenThanikesh (talk) 15:43, 14 September 2016 (UTC)— Preceding unsigned comment added by NarenThanikesh (talkcontribs) 15:30, 14 September 2016 (UTC)

  1. I appreciate the removal of bullet lists
  2. I did not see reference. Yes journals are fine, but books would be better. See WP:RS
  3. Regression testing does not require tools at all, and the tools you listed, Selenium, WATIR, etc. are web tools. What about xunit or other automated functional GUI testing tools? There are also drawbacks to using tools, most famously addressed in James Bach's "Test Automation Snake Oil". Walter Görlitz (talk) 15:58, 14 September 2016 (UTC)
Thank you for pointing out the changes to be made.
  1. Will look into the tools section, What about the drawbacks section? Was the content at fault , because again i read about those drawbacks in journals and referenced them as well. NarenThanikesh (talk) 16:41, 14 September 2016 (UTC)