Look-elsewhere effect

From Wikipedia, the free encyclopedia
Jump to: navigation, search

The look-elsewhere effect is a phenomenon in the statistical analysis of scientific experiments, particularly in complex particle physics experiments, where an apparently statistically significant observation may have actually arisen by chance because of the size of the parameter space to be searched.[1][2][3][4][5]

Once the possibility of look-elsewhere error in an analysis is acknowledged, it can be compensated for by careful application of standard mathematical techniques.[6]

More generally known in statistics as the problem of multiple comparisons, the term gained some media attention in 2011, in the context of the search for the Higgs boson at the Large Hadron Collider.[7]

Use[edit]

Main article: Bonferroni correction

Many statistical tests deliver a p-value, the probability that a given result could be obtained, assuming random coincidence. When asking “does X affect Y?”, it is common to vary X and see if there is significant variation in Y as a result. If this p-value is less than some predetermined statistical significance threshold α, one considers the result "significant".

However, if one is performing multiple tests (“looking elsewhere” if the first test fails) then obviously a p value of 1/n is likely to occur after n tests. For example, an event with p < 0.05 will probably be seen after 20 tests, even if there is no effect whatsoever.[8] In order to compensate for this, you must divide your threshold α by the number of tests n, so a result is significant when p < α/n. Or, equivalently, multiply the observed p value by the number of tests (significant when np < α).

This is a simplified case; the number n is actually the number of degrees of freedom in the tests, or the number of effectively independent tests. If they are not fully independent, the number may be lower than the number of tests.

When the tests are independent, simple multiplication or division by n (called the Bonferroni correction) is only a first-order approximation to the exact Šidák correction.

The look-elsewhere effect is a frequent cause of "significance inflation" when the number of independent tests n is underestimated because failed tests are not published. One paper may fail to mention alternative hypotheses considered, or a paper producing no result may simply be not published at all, leading to journals dominated by statistical outliers.[9]

The effect is particularly important in high-energy physics because of the very large number of tests (many thousands) performed on the same data.

Examples[edit]

  • A Swedish study in 1992 tried to determine whether or not power lines caused some kind of poor health effects. The researchers surveyed everyone living within 300 meters of high-voltage power lines over a 25-year period and looked for statistically significant increases in rates of over 800 ailments. The study found that the incidence of childhood leukemia was four times higher among those that lived closest to the power lines, and it spurred calls to action by the Swedish government. The problem with the conclusion, however, was that they failed to compensate for the look-elsewhere effect; in any collection of 800 random samples, it is likely that at least one will be at least 3 standard deviations above the expected value, by chance alone. Subsequent studies failed to show any links between power lines and childhood leukemia, neither in causation nor even in correlation.[10]

See also[edit]

References[edit]

  1. ^ Lyons, L. (2008). "Open statistical issues in Particle Physics". The Annals of Applied Statistics 2 (3): 887. doi:10.1214/08-AOAS163.  edit
  2. ^ "Synopsis: Controlling for the “look-elsewhere effect”". American Physical Society. 2011. 
  3. ^ Lori Ann White (August 12, 2011). "Word of the Week: Look Elsewhere Effect". Stanford National Accelerator Laboratory. 
  4. ^ Dorigo, Tommaso (2009-10-16). "Supernatural Coincidences And The Look-Elsewhere Effect". Retrieved 2012-10-17. 
  5. ^ Dorigo, Tommaso (2011-08-19). "Should you get excited by your data? Let the Look-Elsewhere Effect decide". CMS Collaboration. 
  6. ^ Gross, E.; Vitells, O. (2010). "Trial factors for the look elsewhere effect in high energy physics". The European Physical Journal C 70: 525. arXiv:1005.1891. Bibcode:2010EPJC...70..525G. doi:10.1140/epjc/s10052-010-1470-8.  edit
  7. ^ Tom Chivers (2011-12-13). "An unconfirmed sighting of the elusive Higgs boson". Daily Telegraph. 
  8. ^ Munroe, Randall (2011-04-06), "Significant", XKCD (882) 
  9. ^ Gritsenko, Vladimir (2009-08-23), "The Journal of (Failed) Replication Studies", Less Wrong, retrieved 2012-06-25 
  10. ^ Palfreman, Jon (1995-06-13), "Currents of fear", Frontline (PBS), retrieved 2012-07-01