Talk:Code coverage

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search
WikiProject Computing (Rated C-class, Mid-importance)
WikiProject iconThis article is within the scope of WikiProject Computing, a collaborative effort to improve the coverage of computers, computing, and information technology on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.
C-Class article C  This article has been rated as C-Class on the project's quality scale.
 Mid  This article has been rated as Mid-importance on the project's importance scale.
WikiProject Computer science (Rated C-class, Low-importance)
WikiProject iconThis article is within the scope of WikiProject Computer science, a collaborative effort to improve the coverage of Computer science related articles on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.
C-Class article C  This article has been rated as C-Class on the project's quality scale.
 Low  This article has been rated as Low-importance on the project's importance scale.
WikiProject Software / Computing  (Rated C-class)
WikiProject iconThis article is within the scope of WikiProject Software, a collaborative effort to improve the coverage of software on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.
C-Class article C  This article has been rated as C-Class on the project's quality scale.
 ???  This article has not yet received a rating on the project's importance scale.
Taskforce icon
This article is supported by WikiProject Computing.

Short circuited evaluation[edit]

C language has short circuit evaluation property. I guess this shall be considered in examples like foo(0,1). In this case second condition is not evaluated at all. — Preceding unsigned comment added by 2A01:111F:4305:3C00:2DBE:582:9150:1AF5 (talk) 09:20, 30 March 2017 (UTC)

Coverage example problem[edit]

In the example: "If the function foo were called with variable bar set to −1, statement coverage would be achieved. Condition coverage, however, would not."

Isn't it contrary? -- 13:40, 13 November 2007 (UTC)

Yeah, I was thinking the same exact thing. Condition coverage WOULD be achieved. However if the number were 1 then condition coverage would be achieved but statement coverage would not. so perhaps this person means to imply that condition coverage does not imply statement coverage.

--Eric —Preceding unsigned comment added by (talk) 22:12, 4 June 2008 (UTC)

I made the change

--Eric —Preceding unsigned comment added by (talk) 22:15, 4 June 2008 (UTC)

But condition coverage DOES imply statement coverage. The example given in the article does not consider that the branch must also be executed for the 'true' case, therefore, having only one test case (bar = -1) does not achieve condition coverage. I have made the edit.

-- Tim —Preceding unsigned comment added by Gibber blot (talkcontribs) 01:48, 20 June 2008 (UTC)

The paragraph about code coverage as a form a debugger doesn't any sense to me. Can someone revisit that part? The rest of the article is great. (talk) 16:52, 25 June 2008 (UTC)

-- Ryan: I would like to suggest that we change the second sentence from 'It describes the degree to which the source code of a program has been tested' to something more like: 'Code Coverage provides a way for developers to determine what percentage of code has had some form of testing, whilst specifically not imply any level of correctness'.

It is easily possible to have code that is not adequately tested and still report 100% code coverage in both statement and branch coverage. Such usage of code coverage can provide a false sense of security to coders and therefore we should not reinforce this. Test coverage indicates the coverage of testing (usually from a functional testing perspective) whilst Code coverage indicates what percentage of code has had some degree of unit testing executed over it, but doesn't imply correctness.

I am happy for someone to reword my suggestion to make it more readable, correct, etc. Distributedlife (talk) 14:10, 12 September 2008 (UTC)

  • Distributedlife, the existing wording does not imply any level of correctness. Perhaps the existing wording needs to explicitly say that testing only shows that the behavior of the code is consistent with the behavior expected by the test that were executed. Derek farn (talk) 14:57, 12 September 2008 (UTC)

  • I agree that it doesn't directly imply correctness, despite my belief that others may derive correctness from such a statement. Perhaps I should have said that code coverage doesn't directly imply test coverage. Code coverage and Test coverage are different. Test coverage is used in relation to the amount of functional testing that has been applied to a set of requirements. To be as accurate as I can be, 'Code Coverage is the percentage of lines of source code that have been executed by unit tests'. I would be happy with that, but what I would be more happy with is that there is some mention that Code Coverate should not be mistaken for Test Coverage. The primary reason for my concern is that I have stumbled across quotes about code coverage that sourced Wikipedia. This propagates the misnomer that code coverage measures the degree of testing that has been applied to source code. a link to the page where I originally sourced the Wikipedia reference

Let me provide an example of why code coverage shouldn't be promoted as way of representing test coverage. Excuse the source code sample:

float reciprocal (const float x)
  return (1.0f / x) ;

If we have a single test that tests the function with a value that is positive. The test will pass, statement and branch coverage will both be 100%. Test coverage is closer to 10% if you factor in positive, negative, zero and values > and < 1.0f. Distributedlife (talk) 16:11, 12 September 2008 (UTC)

Citation needed[edit]

However, a general-purpose algorithm for identifying infeasible paths has been proven to be impossible[citation needed](such an algorithm could be used to solve the halting problem).

the referenced page of the halting problem contains needed citation. Kagamin 15:34, 30 November 2008 (UTC)

External links added to this page is not so good. I miss definitions and examples of diffrent code coverages.

I propose these links for citation or at least external links:

But if you know about some other material with formal definitions available on net, add it instead. But really, I would expect more from wiki than one sentence about each type. Add formal definitions and examples. --Blacklily (talk) 10:03, 9 October 2009 (UTC)

We generally don't link to blogs, see the external link guideline. - MrOllie (talk) 13:49, 9 October 2009 (UTC)

First link is not blog. But I do not care what links ou add If it has valuable informations. In current external links there are tree links. First link

is very good and usefull.

Second link shoud be deleted, it is not very useful, no definitions of coverages.

Third link should be deleted too. It is just link to some pdf that is not publicly available.

So only first link is useful.

Delete these bad links and let's find someting useful. I found this:

a I found useful and simple table about coverages in next link, too:

-- (talk) 09:52, 14 October 2009 (UTC)

Insure++: does this belong?[edit]

In looking at Insure++'s site, I see no mention of code coverage. It appears to be a memory leak detector.

It also measure code coverage. CodeCurmudgeon (talk) 15:44, 12 July 2018 (UTC)

More Coverage Programms[edit]

In my opinion "NCover" should be on the list of available Software Code-Coverage Tools. ( [1]) (talk) 07:42, 3 March 2009 (UTC)

Software Code Coverage Tools?[edit]

Why does this page list hardware code coverage tools but not software code coverage tools? Surely hardware-based tools are more esoteric and less used? A table would be good showing tool, license, link, environment/OS, etc? Some to include are Cobertura, EMMA, NCover, rcov. —Preceding unsigned comment added by (talk) 12:56, 5 August 2009 (UTC)

Software Code Coverage Tools were all removed by User:MrOllie due to this edit: [2] - reason: Vendor link sections not allowed per WP:EL. IMHO not a good decision to remove a complete paragraph because there are some vendor links in it. --> I've again added all tools without vendor links and added a reminder for not adding links again (what MrOllie should have done in the first place) --Sebastian.Dietrich (talk) 18:22, 25 August 2009 (UTC)

Coverage Metrics[edit]

Added both Linear Code Sequence and Jump (LCSAJ) & JJ-Paths as coverage metrics. Nat Hillary 18:50, 11 November 2009 (UTC) —Preceding unsigned comment added by Nat hillary (talkcontribs)

'Basic coverage criteria' a mess[edit]

In the bullet-point list, we now have 'Decision coverage' as well as 'Condition coverage' both listed twice, and 'branch coverage' and 'predicate coverage' thrown in there too. All these terms seem to refer to the same thing. There is also no mention in the multiple examples of anything involving multiple branches, like case statements, elseif or try-catch-catch. Can somebody who has access to the cited source (Glenford J. Myers (2004): The Art of Software Testing) tidy it up to be more general and more logical, or are we going to have to sort it out from first principles ourselves? --Nigelj (talk) 11:02, 21 June 2010 (UTC)

I cleaned-up this now. This section should really contain only basic criteria. Other more advanced criteria should go to other sections. Andreas Kaufmann (talk) 09:56, 9 February 2014 (UTC)

Hardware Coverage Tool Providers -- why is this present?[edit]

I don't see why hardware tool vendors are listed here, and software tool vendors are not. Some earlier discussin here suggests that *links* to tools/vendors/products (presumably tool/vendor websites were meant) are not good. I don't see why a link to a vendor's company name listed under Wikipedia is different. If you allow this, companies that are big enought to get listed in Wikipedia get an unfair advantage, since there's a one-to-one relationship between "wikilink to vendor" and link to vendor commecial website.

Either let people list all the tools they know about, or insist that none of them are reasonable. —Preceding unsigned comment added by (talk) 03:52, 23 August 2010 (UTC)

Incorrect MC/DC test set[edit]

For MCDC, the independence of a conditions effect on the decision is achieved by holding all other conditions constant while toggling that condition between true and false.

In the test set suggested to satisfy MC/DC, there is no MCDC pair for condition 'c' where the values for 'a' and 'b' remain constant:

   a=false, b=false, c=true
   a=true, b=false, c=true
   a=false, b=true, c=true
   a=true, b=true, c=false

I suggest changing the last row of the set to:

   a=false, b=true, c=false

Alonergan76 (talk) 08:09, 2 August 2012 (UTC)

Weird Pascal example[edit]

The article says

In languages, like Pascal, where standard boolean operations are not short circuited, condition coverage does not necessarily imply decision coverage. For example, consider the following fragment of code:

if a and b then

Condition coverage can be satisfied by two tests:

  • a=true, b=false
  • a=false, b=true

However, this set of tests does not satisfy decision coverage as in neither case will the if condition be met.

Except that makes no sense. In all languages, if at least one of "a" and "b" aren't true, then "a and b" ("a && b", "a and then b", "a andalso b") won't be true, whether or not it uses short-circuit evaluation. That's what "and" means.--Prosfilaes (talk) 09:10, 20 February 2013 (UTC)


The article cites Glenford J. Myers (2004). The Art of Software Testing, 2nd edition, as source for the various types of coverage. However, Myers doesn't talk about function coverage. — Preceding unsigned comment added by (talk) 11:14, 29 February 2016 (UTC)

Code coverage improperly described as test coverage[edit]

Strangely this article describes code coverage as test coverage in the lead, but they are not the same thing. Code coverage means "what code got executed" while test coverage means "what requirements got tested". I'll go track down some good sources, if anyone has suggestions on sources or improved wording let me know. CodeCurmudgeon (talk) 20:32, 11 July 2018 (UTC)

w:Martin Fowler says ... no. [3] says they're the same. "C. Prause, J. Werner, K. Hornig, S. Bosecker, M. Kuhrmann (2017)" (ref #12 in the article) uses test coverage for this. It's probable that there's multiple meanings for the same words out there. I like "test coverage" = the amount that the tests cover.--Prosfilaes (talk) 06:11, 12 July 2018 (UTC)
Yes, I know that Fowler says the opposite and we may have to point that out, but most sources make the distinction which is important. Industries standards such as MISRA, JSF, and DISA-STIG require code overage which they then describe as I've mentioned above - meaning what code is covered. While test metrics are oriented toward what requirements were covered. You cannot use the same name to mean both. If we just say "The amount that the tests cover" what are we talking about - code that code executed, or requirements that were tested? CodeCurmudgeon (talk) 14:43, 12 July 2018 (UTC)
We're talking about what the tests cover. It could be decision points, it could be lines of code, it could be functions, or lines of functional code or requirements. If you want to talk about "of code" or "of requirements", you'd say "test coverage of code" or "test coverage of requirements". That's my personal preference, but it's pretty clear to me that there's no standard usage here.--Prosfilaes (talk) 20:53, 12 July 2018 (UTC)
You might say "of code" or "of requirements" but that's not what any of the standards do. It's not the engineering definition of the terms. CodeCurmudgeon (talk) 14:50, 13 July 2018 (UTC)
Brian Marick has a pretty good explanation - that "test coverage" is an umbrella" and "code coverage" is a type of "test coverage". CodeCurmudgeon (talk) 15:48, 12 July 2018 (UTC)
But that's different from your distinction. That's my point, that your distinction, your way of using the words isn't the "correct" way, it's just one alternative.--Prosfilaes (talk) 20:53, 12 July 2018 (UTC)
Actually IEEE defines test coverage the same way It's not just my distinction. While many bloggers and devlopers just use the terms in various ways, they have actual meanings and definitions, especially when you get into compliance issue, like MISRA< JSF, FDA, DISA, as I mentioned. The terms have actual separate definitions. CodeCurmudgeon (talk) 14:36, 13 July 2018 (UTC)
If you're into compliance issues, a bunch of words have meanings they don't have in real life. Words mean what people use them to mean, and it's clear that Martin Fowler uses them one way, Brian Marick uses them another way, and IEEE doesn't define "code coverage" at all. And the very IEEE document you offer says, on the second page of the PDF, second paragraph, use of this is voluntary, there are other ways to do this, and this document is out of date. (It says "when this document is more than five years old", "its contents ... do not reflect the state of the art", and this document is almost thirty.)
Word mean what people use them to mean. I'm not arguing that there doesn't need to be clarification here, but I also find it clear that English speakers use these words to mean a variety of different things, and therefore there's no one "actual" meaning. --Prosfilaes (talk) 20:39, 13 July 2018 (UTC)
I get that people are using the terms differently, but they are actually defined by standards bodies, whether compliance is required or not. And interestingly, the different bodies are using the same definitions. So yes, it's important to mention that some people use the terms differently, but that doesn't negate the meanings of the words. Words do not in fact mean what people want them to mean, the only way to communicate is to have the same defintion - that's the whole point of the standards organizations. I know that IEEE is old, but core definitions do not in fact change. Practices and tools do. CodeCurmudgeon (talk) 23:49, 13 July 2018 (UTC)
"Words do not in fact mean what people want them to mean". Then how are we communicating? Why are you bothering trying to communicate in English, which has no standards body governing it? Try French. If you want to communicate, you define your words where necessary, like this article does; declaring that you're using the definitions of an expensive proprietary standards document doesn't in fact help you communicate with most people.
We should take the IEEE as gospel, except when they say that the document is out of date?--Prosfilaes (talk) 01:44, 15 July 2018 (UTC)
I don't think a philosophical discussion on the nature of language is necessary to our task here. Namely that in computer science terms do indeed have definitions, and those definitions are both helpful and necessary to make sure that when one group says "we have achieved xxx" that another group can know what that represents. Standards bodies and technical organizations have contributed to these efforts in the form of ISO, IEEE, ASQ, etc. My goal is to represent here what the terms actually mean so that those learning about the topic will be informed rather than misinformed. CodeCurmudgeon (talk) 14:36, 16 July 2018 (UTC)