Talk:Bonferroni correction

Discussion of Holm-Bonferroni is inappropriate

If nothing else, this paragraph requires citations. Holm-Bonferroni is one of many sequential Bonferroni-type approaches to controlling the familywise error rate in a set of tests. Among these many approaches, it is not even particularly powerful---Hochberg's approach is, for example, guaranteed to be no less powerful and is in many cases more powerful. The paragraph should be changed to reference two classes of more pwoerful tests: (1) sequential Bonferroni-type tests that control the familywise error rate, and (2) Benjamini-Hochberg type tests that control the false discovery rate. Specific reference to an individual test such as the Holm's is out of place, as it appears to provide preference to one of many alternatives. —Preceding unsigned comment added by 71.48.130.128 (talk) 13:21, 27 October 2009 (UTC)

Thank you for your suggestion. When you believe an article needs improvement, please feel free to make those changes. Wikipedia is a wiki, so anyone can edit almost any article by simply following the edit this page link at the top. The Wikipedia community encourages you to be bold in updating pages. Don't worry too much about making honest mistakes—they're likely to be found and corrected quickly. If you're not sure how editing works, check out how to edit a page, or use the sandbox to try out your editing skills. New contributors are always welcome. You don't even need to log in (although there are many reasons why you might want to).--mcld (talk) 19:27, 27 October 2009 (UTC)

Šidák correction should be moved to its own article

Though related, the Bonferroni and Šidák tests rely on different assumptions and lead to generally different results. It is confusing to discuss both on a page labeled "Bonferroni correction." —Preceding unsigned comment added by 71.48.130.128 (talk) 13:24, 27 October 2009 (UTC)

This article is incorrect

It says "one way of maintaining the familywise error rate is to test each individual hypothesis at a statistical significance level of 1/n times what it would be if only one hypothesis were tested", i.e., alpha_corrected = alpha/n. The formula given is *another correction*, which is "applicable for two-sided hypotheses, multivariate normal statistics, and positive orthant dependent statistics, it is not, in general, correct (Shaffer 1995)". See http://mathworld.wolfram.com/BonferroniCorrection.html —Preceding unsigned comment added by Leonardo.hamilton (talk • contribs) 10:58, 20 February 2009 (UTC)

Extra references

I have added extra references to this article so that people can conveniently find out more about this method. I have also added a direct link to Thomas Perenger's critique of the Bonferroni method, as it is cited in the three other references I have added. Michael Glass 13:23, 9 June 2007 (UTC)

I'm not sure about the point of adding this article by Perenger. It is a seriously flawed article in that you most definitely are not just rejected the global null hypothesis, but in fact also the individual hyphothesis. All of Perenger's arguments along of the lines of "And type II errors are no less false than type I errors" are rather pointless, if one pretends to perform multiple tests at a fixed type I error rate. One a whole the author just does not understand the Bonferroni adjustments. A much more reasonable criticism would be that Bonferroni-Holm is uniformly more powerful, but for example it's much harder to obtain valid Confidence intervals for the Bonferroni-Holm procedure (there's a recent paper on how to do it by Bretz et al., but these confidence intervals are surprisingly not always contained within the Bonferroni ones). Baselbonsai 00:01, 16 August 2008 (CET).

Clarification needed

There is a need to clarify how the outcome of the multiple tests is judged .... "at least one test says reject", or "all tests say reject". Melcombe (talk) 17:06, 29 April 2008 (UTC)

Additive

There was some notation implying that you specify the tests to be of size alpha/n, when they should be of size 1-(1-alpha)^(1/n)... That is, one minus the chance of not rejecting each test. Why would these things be written as 1-(1-alpha/n)? Brusegadi (talk) 08:20, 14 February 2009 (UTC)

-Not developed by Bonferroni

Bonferonni correction was not developed by Bonferroni - it was developed by Dunn, based on a proof of an inequality. However, the proof of the inequality was not done by Bonferroni either, he extended it. I'll edit this when I have the time. Bonferroni correction is based on Boole's inequality - this is discussed in the article Stigler's law of eponymy by Steven Stigler. —Preceding unsigned comment added by Jeremymiles (talk • contribs) 05:04, 18 March 2010 (UTC)

Missing citation

The statement "Second, in certain situations where one wants to retain, not reject, the null hypothesis, then Bonferroni correction is non-conservative." could do with a citation Worik (talk) 00:47, 29 March 2010 (UTC)

I'd say that 'conservative', in its statistical sense, means 'rejecting the null hypothesis when it is true less often than indicated by the nominal p-value or specified Type I error rate', but i don't have a citation for that to hand either (other than conservatism (disambiguation), which is hardly a WP:reliable source). I may try to find one... Qwfp (talk) 19:45, 29 March 2010 (UTC)

"Lou Jost" test?

The only source for the "Lou Jost" test is Lou Jost's own website - his suggestion does not seem to have been subject to any peer-review. I have removed the reference and added tags; if someone cannot dig up a good source, I will remove that section. Angio (talk) 18:43, 15 November 2010 (UTC)

Confidence intervals

From the current article:

“Bonferroni correction can be used to adjust confidence intervals. If we are forming $m$ confidence intervals, and wish to have overall confidence level of $1-\alpha$, then adjusting each individual confidence interval to the level of $1-{\frac {\alpha }{m}}$ will be the analog confidence interval correction.”

This claim should be sourced.Vegard (talk) 17:06, 25 September 2012 (UTC)

The paper by Dunn in 1959

The article says Dunn have wrote two papers in 1959 and 1961, but no paper in 1959 is cited in References section. I suppose the 1959 paper is - Dunn, O. J. (1959) Confidence intervals for the means of dependent, normally distributed variables. Journal of the American Statistical Association, 54, 613-621. Am I correct?Francesco Nagoya (talk) 03:47, 15 February 2013 (UTC

In Plain English

Can we add the sentence "In order to do a Bonferroni Correction, divide your significance threshold $\alpha$ by the number of hypotheses being tested" somewhere, preferably in the introduction? I know the article says this, but anyone who isn't a mathematician is going to have to read closely in order to extract that piece of information. (Assuming it is correct).

In general this article is extremely technical for anyone who isn't a statistician. These corrections have applications (and are widely used, in fact) in other areas of sciences and significance testing. It would really be helpful if the article had a less technical section - which obviously needs to be written by a statistician in order for the article to remain accurate. Sadly, I am not a statistician and hence I cannot do this myself. 130.209.157.26 (talk) 10:16, 22 October 2014 (UTC)

Agreed 100%. The introduction needs a specific example at the outset; not just for the technique, but also justifying why alpha is linearly divided by the number of hypotheses (or not multiplied for that matter). The article seems written to impress those who know, not those who do not. ELeP PerthAU (talk) 14:40, 25 November 2015 (UTC)ELeP 20151125

Dubious

The derivation of the correction substitutes having a conservative p-value for committing a type I error. The outcome is the same, but the presentation and therefore strictly speaking the logic is erroneous. I was thinking about how to correct it, but perhaps it should be handled by somebody more familiar with the subject matter's conventional presentation. — RFST (talk) 08:37, 12 August 2014 (UTC)

Source removal by IP

In this edit by an IP a WP:RS was removed as being "full of misinformation". I reverted the edit , because I want a full discussion of what the supposed misinformation is.--Wuerzele (talk) 17:28, 17 September 2016 (UTC)

I don't know if there is a Wikipedia manual of style on the issue, but I think that if something appears in a respectable journal and is older than a year or so then the presumption is that it is reasonable, unless there are subsequent articles indicating to the contrary. That there are posted responses that disagree with the article is not sufficient; can we find a peer-reviewed article that disagrees with Perneger 1998? 𝕃eegrc (talk) 13:22, 22 September 2016 (UTC)

Thanks for chiming in 𝕃eegrc, the problem is that User:23.242.207.48 never discusses his edits on talk pages, if you review their contributions. They think explaining in edit summaries is sufficient! This user has been warned of edit warring before, so it is unwillingness to learn.

The IP which traces to UCLA has removed the source twice now

1. 12:50, 17 September 2016
2. 05:45, 18 September 2016
3. and on 03:57, 22 September 2016 I suspect they used their home IP as User:131.179.36.78 for the latest reversal, which traces to residential customers in Orange county amongst others.- The edit summary was the same type: "Not unexplained at all".Wuerzele (talk) 19:08, 22 September 2016 (UTC)

The authors of the rapid response, Bender and Lange, later published a peer-reviewed paper on the same issues in the Journal of Clinical Epidemiology that Google Scholar says has been well-cited: [1]. Maybe adding this to 'Further reading', rather than removing Perneger, could be an acceptable compromise solution? Qwfp (talk) 20:04, 22 September 2016 (UTC)

I haven't looked at Bender and Lange but, assuming that it confronts the ideas of Perneger 1998 as Qwfp says, I would include Bender and Lange in the Further Reading list, as suggested. I would also note right in the Perneger 1998 citation something like "though see Bender and Lange". What do you think? 𝕃eegrc (talk) 20:33, 22 September 2016 (UTC)

I'm not going to respond to Wuerzele's ad hominem attacks, other than to note the irony of his complaints about edit warring and the creepiness of his attempts to figure out where I live. Let's try to avoid that kind of behavior, which is better suited for a YouTube comment section than an encyclopedia, and which I'm sure violates some Wikipedia policy or another. Instead, let's focus on the issue at hand, which is how to build a better article. With that in mind, we should ask, "what does the Perneger link add to the article?" It contains no unique correct information whatsoever, and plenty of wrong information. As one researcher observed, it "consists almost entirely of errors" (http://www.bmj.com/content/318/7176/127.2). Therefore, recommending it as a reference DOESN'T HELP AND MIGHT HURT.

One editor has asked for a peer-reviewed source refuting Perneger, which is a reasonable request. A good example of a peer-reviewed article that points out Perneger's cluelessness on the topic is the Goeman & Solari paper (from Statistics in Medicine, a respected statistical journal) that is cited in this article. It points out two very fundamental "misunderstandings" at the heart of Perneger's arguments: number 1 that Bonferroni assumes independence, and number 2 that Bonferroni only protects in the case of the global null. It's not that Perneger's opinions on these things are controversial, it's that he's objectively, mathematically wrong and doesn't seem to understand how Bonferroni correction relates to probability at its most basic level. This has been pointed out in the Goeman & Solari paper, in the Bender & Lange response in BMJ, in Aickin's letter to the editor mentioned above, and elsewhere. So the only conceivable way Perneger's paper could be usefully referenced in this article would be as an example in a section on common misunderstandings about the Bonferroni correction. To recommend Perneger's paper as either an expert opinion on the subject or as a debatable but relevant and mathematically accurate perspective is misleading. — Preceding unsigned comment added by 23.242.207.48 (talk) 01:04, 24 September 2016 (UTC)

I have not read the articles cited by 23.242.207.48 but, assuming that they are correctly reflected in these descriptions, I see no reason Perneger should be listed as Further Reading. However, I worry that simply removing the citation opens the article to (well meaning) edits in the future that restore it. To address this issue, I would support a short section on common misunderstandings. 𝕃eegrc (talk) 12:01, 26 September 2016 (UTC)

External links modified

Hello fellow Wikipedians,

I have just modified one external link on Bonferroni correction. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:

• Added archive https://web.archive.org/web/20110716071000/http://www.silicongenetics.com/Support/GeneSpring/GSnotes/analysis_guides/mtc.pdf to http://www.silicongenetics.com/Support/GeneSpring/GSnotes/analysis_guides/mtc.pdf

When you have finished reviewing my changes, please set the checked parameter below to true or failed to let others know (documentation at {{Sourcecheck}}).

This message was posted before February 2018. After February 2018, "External links modified" talk page sections are no longer generated or monitored by InternetArchiveBot. No special action is required regarding these talk page notices, other than regular verification using the archive tool instructions below. Editors have permission to delete these "External links modified" talk page sections if they want to de-clutter talk pages, but see the RfC before doing mass systematic removals. This message is updated dynamically through the template {{source check}} (last update: 5 June 2024).

• If you have discovered URLs which were erroneously considered dead by the bot, you can report them with this tool.
• If you found an error with any archives or the URLs themselves, you can fix them with this tool.

Cheers.—InternetArchiveBot (Report bug) 20:01, 5 November 2016 (UTC)

Change of Title from Bonferroni to Dunn-Bonferroni

@Groceryheist: The change of title is inappropriate and inconsistent with article naming criteria. The new title, and accompanying article text change, are less recognizable (the old name is far more common in literature), less natural (few people are searching for the alternate name), less concise (without resolving any ambiguity), and less consistent (the old name was near universal throughout Wikipedia). Many things in science have unusual or "incorrect" names; It is not the role of Wikipedia to assert non-neutral, less common names as if they were consensus to reflect what some users think things "should be called". Echocancellation (talk) 16:51, 27 June 2020 (UTC)

@Groceryheist: Lets turn this around from an accusation to a question: Why do you think that this is appropriate and consistent? Could you, for example, point to authoritative literature that uses this name? I was pointed on Twitter to the APA dictionary as one source, but you may be aware of more. Note that it is not necessarily appropriate to only look at the historically most frequently used name. effeietsanders 18:58, 27 June 2020 (UTC)

@Effeietsanders: The new name is simply to keep the Wikipedia up to date. While the old name Bonferroni-correction may be more widely used in the academic literature it is quickly becoming out of date and will soon be overtaken by Dunn-Bonferroni. The new name gives proper attribution to the originator of the technique. The APA dictionary reflects this development and Wikipedia should too. Groceryheist (talk) 19:28, 27 June 2020 (UTC)

@Groceryheist: Saying that it "will soon be overtaken" is a naked assertion; The proposed renaming appears intended to motivate this change, not merely reflect it. I attempted to compare the two terms with both Google Search Trends and Ngram Viewer, but "dunn-bonferroni" was so rare that it wasn't even possible to make a chart. Second, of the first several pages of Google search results I skimmed for multiple comparisons corrections, not a single one used the proposed naming. Finally, although not a precise tool, Google Scholar search results, when limited to the year 2019, still show the proposed name has about 2% of the hits as the existing name (under 700 vs 29,600). Echocancellation (talk) 19:58, 27 June 2020 (UTC)

"Bonferroni correction" does seem more widely used/recognized historically; however, changes in usage standards may be recent (e.g., the APA Dictionary example provided by effeietsanders) and usage without Dunn's name may continue due to ignorance and path-dependency more than anything else (perhaps I'm just speaking for myself there). With new and/or more recent terminology changes, Google ngrams or search trends may not resolve the question decisively—see: WP:Namechanges on extra weight to reliable sources as/after changes occur. Within Google scholar results, around 50% of the uses of "Dunn-Bonferroni correction" occur within the last 5 years, suggesting that Groceryheist's assertion may not be quite as naked as it looked. At a minimum, the historical absence and more recent inclusion of Dunn's name in standard, scholarly sources should be addressed explicitly in the article and via redirects. The invisibility or erasure of Dunn's work should be a part of the encyclopedic record. Aaron (talk) 21:16, 27 June 2020 (UTC)

Just to say that the APA Dictionary cited above is not correct. Nobody has to believe me -- just look at the paper by Dunn (1961) to confirm. Ideally we would revert all changes made today, then add to the section (already existing) on confidence intervals that for that use the method can be named Dunn-Bonferroni. There are too many wrongs in the world already, we don't need to compound to it... Winkler (talk) 22:35, 27 June 2020 (UTC)

Requested move 27 June 2020

Dunn–Bonferroni correction → Bonferroni correction – The recent move by user Groceryheist was done unilaterally, with no discussion, and citing no sources, to reflect what some people think this statistical procedure "should have been called", even though it is by far a less common name, both elsewhere on Wikipedia and in published educational material. A quick search of journal articles would reveal that the old name was in more common use, and the one people would likely be searching for this article by. Echocancellation (talk) 17:27, 27 June 2020 (UTC)

@Echocancellation: See my comments above re: some arguments for/against the move and naming. I provisionally support having the article at Bonferroni correction, but with a redirect pointing there from Dunn–Bonferroni correction and with content edits to address that both names are in current usage supported by reliable sources. Aaron (talk) 21:22, 27 June 2020 (UTC)

@Echocancellation: :@Aaronshaw: The change is incorrect. This started from a single tweet who claimed an anonymous reviewer had said it should be called Dunn-Bonferroni, but if one examines the papers will see that this makes no sense. Dunn (1961) discussed using Bonferroni (1936) to correct confidence intervals (as opposed to point estimates). There is a 25 years difference. Dunn found an application for the Bonferroni method, but didn't really develop the Bonferroni method nor proved it. However, the related Dunn-Sidak method should be (correctly) named after Dunn, since that was a novel contribution (paper published in 1958). Winkler (talk) 21:43, 27 June 2020 (UTC)

@Winkler: Just to clarify: are you suggesting that all instances that Aaron refers to above, are based on that Tweet? effeietsanders 21:45, 27 June 2020 (UTC)

@Effeietsanders: I don't know where Aaron or anyone is coming from. What I know is that there is a small wave of support that formed today on Twitter [[2]] for renaming it to Dunn-Bonferroni. But this only holds true if one is taking about correcting confidence intervals. The general Bonferroni correction method was already widely popular when Dunn published her paper in 1961. She calls it Bonferroni in the paper herself! Now, for Dunn-Sidak, that is correct, and it should continue to be called Dunn-Sidak. Dunn-Bonferroni is not correct unless perhaps for the specific use of Bonferroni to correct confidence intervals in the comparison among means. Winkler (talk) 21:55, 27 June 2020 (UTC)

@Winkler: Just because there's a wave of support today because someone tweets about it, doesn't mean it's any less true. As I linked, and Aaron indicated, there are more sources that have been using this name. It's fine to disagree with that being sufficient, but please don't disregard it as 'a single tweet'. effeietsanders 22:00, 27 June 2020 (UTC)

@Effeietsanders: ok, np! However, the article title should be reversed. Maybe a section discussing the use of Bonferroni for CIs could be added, in which Dunn's contribution is included. And of course, Dunn-Sidak should continue be called as such. Winkler (talk) 22:07, 27 June 2020 (UTC)

@Winkler: It has a section dedicated to the extension proposed by Dunn to confidence intervals thisbugisonfire (talk) 22:13, 27 June 2020 (UTC)

@Thisbugisonfire: Excellent! So now the article title can be reversed back, since the method was originally (in 1936) proposed by Bonferroni. It is wrong to adjust p-values or correct the level of a test using this method and call it "Dunn-Bonferroni". As it is right now, Wikipedia is providing wrong information. Winkler (talk) 22:18, 27 June 2020 (UTC)

I think something is off. The Wiki article is about the correction for p-values (Bonferroni formulated that one in 1936, according to wiki). Dunn procedure is about confidence intervals (1958-1961). Albeit related, they are different things. Perhaps add an article about the Dunn procedure and cross-link both? thisbugisonfire (talk) 21:50, 27 June 2020 (UTC)

For example, see the quote in Olive Jean Dunn. thisbugisonfire (talk) 21:53, 27 June 2020 (UTC)

Just to make sure people are aware of this, there is also the article Holm–Bonferroni method and there's Boole's_inequality#Bonferroni_inequalities. effeietsanders 22:04, 27 June 2020 (UTC)

@Effeietsanders: The Holm-Bonferroni correction is different to the Bonferroni correction, it's even mentioned in the article. thisbugisonfire (talk) 22:07, 27 June 2020 (UTC)

I wasn't trying to claim that it was the same, sorry if that was accidentally implied. Just noticed that dewiki has a single article about both (right or wrong), and that it's just good to be aware it exists (above there were some references to the 'method') in case someone wants to combine both concepts - I wasn't sure if you were aware of it. I overlooked myself that it was already linked in the article. effeietsanders 22:13, 27 June 2020 (UTC)

Unfortunately, I don't agree with this edit and moving this page. I believe the cause for these recent edits is this tweet, but unfortunately it's wrong. The Dunn-Sidak correction is not the same as the Bonferroni correction, and one of the sources cited for this correction is an APA dictionary link, but it's clear the APA dictionary link is incorrect, just look at the original papers. I think the original intentions to give due credit are noble but the actual claims are incorrect and based on incorrect sources. Lesslikely (talk) 22:27, 27 June 2020 (UTC)

Also worth noting that the Dunn-Šidák correction has its own page. I'd argue that the title and URL should be completely renamed to Dunn-Šidák from Šidák, but first, this edit should be sorted out Lesslikely (talk) 22:31, 27 June 2020 (UTC)

• Comment - it doesn't look like anyone has really started discussing this move and this discussion has mostly been linking to Twitter opinions. Somebody dig up a better source for this, whether sources call it "Bonferroni" or "Dunn-Bonferroni", then we can reach consensus. If any more experienced editors are skimming through , you might want to participate on this one. Ed6767 talk! 01:01, 28 June 2020 (UTC)

Groceryheist, you moved this page, but you didn't give a reason past "Give credit to Olive Dunn". However, many scholarly articles I've seen (via a quick Google) only call this the "Bonferroni correction" but also mention how it was "first attributed to Olive Jean Dunn", not to mention over 2 million results for the original title, and only 4,000 for the new one - I strongly support the move back to Bonferroni correction per WP:COMMONNAME - I find little references to this as the Dunn–Bonferroni correction, but also think that Dunn should totally be attributed. Ed6767 talk! 01:17, 28 June 2020 (UTC)

indeed, I think Dunn should absolutely get credit for her contributions, but that should be on the Dunn-Šidák page, not the Bonferroni page Lesslikely (talk) 23:27, 27 June 2020 (UTC)

- Given the inaccuracies of the claims, and the sources used to make the claim, I strongly support moving back the page immediately if we want to reduce as much confusion as possible. Lesslikely (talk) 14:29, 28 June 2020 (UTC)

@Echocancellation: @Aaronshaw: @Winkler: @Ed6767: @Effeietsanders: @Thisbugisonfire: @Groceryheist: Can we please get this resolved? User @Groceryheist: should also very seriously consider looking at Wikipedia editing guidelines and etiquette before making any further changes anywhere else on the site Lesslikely (talk) 14:56, 28 June 2020 (UTC)

• Support: renaming the page back to Bonferroni, and also creating a page for the Dunn procedure (which is really important and was done by Dunn, as all references show), with cross-linking between both. I get the idea of the original editors, people really credit Bonferroni for the CI correction, but it's due to Dunn, really, even though the principle is the same. The page, however, is about the correctiong for the significance level (or, alternatively, the p-values) thisbugisonfire (talk) 15:14, 28 June 2020 (UTC)