Talk:Regression toward the mean/Archive 3

I need a proof of:

For any c in [a,b]:
• Pr( [Sum (over all i such that y_i > c) e_i] > 0) > 0.5
• Pr( [Sum (over all i such that y_i < c) e_i] < 0) > 0.5

This is the obvious generalization of his formulation. I assumed wrongly that he knows how to generalize his formulation. What AaCBrown provided is rubbish, purporting to prove that "for y_j > c, Pr(e_j > 0) > 0.5", something entirely different. His "proof" is:

Consider any point c in the range [a,b]. Consider any j. Unconditionally, e_j is equally likely to be positive or negative. If x_j > c, if e_j > 0 then Pr(y_j > c) = 1. If e_j < 0 then Pr{y_j > c) < 0.5. If x_j < c, if e_j > 0 then Pr(y_j > c) > 0 and if e_j < 0 then Pr(y_j > c) = 0. So either way, for y_j > c, Pr(e_j > 0) < Pr(e_j < 0), so Pr(e_j > 0) > 0.5

This "proof" does not even prove what it claims to prove. I have gone through this trash, and it is totally worthless. (It is an excellent exercise for gifted ninth graders to debunk.) The most egregious error is this "so", equating Pr(e_j > 0) to Pr(y_j > 0) [This gross error can be spotted by gifted ninth graders easily]:

So either way, for y_j > c, Pr(e_j > 0) < Pr(e_j < 0), so Pr(e_j > 0) > 0.5.--Palaeoviatalk 01:43, 2 August 2010 (UTC)

There are several other elementary mistakes. I recommend this, seriously, as an exercise for gifted eleventh graders to debunk point by point. (See my "Guide to debunkers" below.)--Palaeoviatalk 02:16, 2 August 2010 (UTC)

I am convinced that I have been absolutely right not to trust the arguments on RTM of someone whose mathematical sophistication, mathematical maturity, and muddleheadedness are reflected in such a proof as this.--Palaeoviatalk 02:23, 2 August 2010 (UTC)

The exercise is to debunk, in detail, the following "proof".

Claim Let X = {x_1, x_2, . . ., x_n} be any set of unknown points. Let E = {e_1, e_2, . . .,e_n} be unknown i.i.d draws from a distribution with median zero and support over the entire real line. We observe only Y = {x_1 + e_1, x_2 + e_2, . . ., x_n + e_n}. The minimum value of y is a, the maximum value is b. Let c be any value in the range [a,b].

For all j such that y_j > c, Pr(e_j > 0) > 0.5.

(Though greater clarity is possible, I have preserved the original phrasing of the Claim.)

Invalid Proof (for debunking): It is in the citations, and it is as trivially obvious as the first proof. In fact, it's the same argument and I can do better, I can prove it for every point, not just the sum. However, again, I'm just reporting from the sources, none of this is my original work.

Consider any point c in the range [a,b]. Consider any j. Unconditionally, e_j is equally likely to be positive or negative. If x_j > c, if e_j > 0 then Pr(y_j > c) = 1. If e_j < 0 then Pr{y_j > c) < 0.5. If x_j < c, if e_j > 0 then Pr(y_j > c) > 0 and if e_j < 0 then Pr(y_j > c) = 0. So either way, for y_j > c, Pr(e_j > 0) < Pr(e_j < 0), so Pr(e_j > 0) > 0.5

(Note: see subsection Correct Notation below.)

To start gifted eleventh graders taking up this challenge off, let me analyze the beginning of the "proof":

First. note the bold phrases. They illustrate proof by intimidation. Always refuse to submit to such a proof tactic. Now we examine:

If x_j > c, if e_j > 0 then Pr(y_j > c) = 1.

Remember that the only probability space is that of error E. So "If x_j > c, if e_j > 0" means that x_j, error e_j, y_j are all known, and no more uncertainty remains. "y_j > c" is true. You should say "y_j > c" (a certain fact). It is wrong to say "Pr(y_j > c) = 1". Pr should always refer to the probability space in question. Now the next sentence,

If e_j < 0 then Pr{y_j > c) < 0.5.

Now this is confusing. If e_j is known, then y_j is also known, and (y_j > c) should be either true or false. "Pr{y_j > c) < 0.5" makes no sense. Is this Pr{y_j > c) the probability before e_j is known? Is so, then "Pr(y_j > c) = 1" in the earlier sentence must also refer to the probability before e_j is known. But how can the earlier sentence say "Pr(y_j > c) = 1" (i.e. before e_j is known, it is certain,with probability 1, that (y_j > c))? It is plainly false. So we are in a major notational and conceptual muddle here. Very sloppy thinking is exhibited here. Try to avoid such laziness and sloppiness of thought. Such sloppy thinking can lead to "discoveries" such as "0=1".

I'll leave the rest to you. You can have great fun debunking this "proof".

It is a good exercise in clear and rigorous mathematical reasoning.--Palaeoviatalk 05:08, 2 August 2010 (UTC)

Correct Claim

We have above this Claim:

Claim Let X = {x_1, x_2, . . ., x_n} be any set of unknown points. Let E = {e_1, e_2, . . .,e_n} be unknown i.i.d draws from a distribution with median zero and support over the entire real line. We observe only Y = {x_1 + e_1, x_2 + e_2, . . ., x_n + e_n}. The minimum value of y is a, the maximum value is b. Let c be any value in the range [a,b].

For all j such that y_j > c, Pr(e_j > 0) > 0.5.

The "proof" was a mess. The question remains: Is the claim true? Is there a valid proof? The answer is "No". The Claim is false. No valid proof exists for a false claim.

(Note: See subsection Correct Notation below.)

It is straight forward. The intention is to assert something about Pr(e_j > 0), for all j such that y_j > c.

Values in E are unknown. Consider any j (a particular j) such that y_j > c (we don't know which numbers qualify as such j yet). What is Pr(e_j > 0)?

Simple. Because the median of the error (E) distribution is 0, Pr(e_j > 0) = .5 . (This is in fact true of any j in [1,n].)

The correct (trivial) claim is therefore:

For all j such that y_j > c, Pr(e_j > 0) = 0.5--Palaeoviatalk 16:00, 2 August 2010 (UTC)

Correct Notation

I need to highlight here a notational issue. I wrote statements such as "Pr(e_j > 0) = 0.5" because AaCBrown wrote with this wrong notation, and I wanted to focus only on the essential conceptual errors in his prrof. e_j is a specific value, and no probability space exists with respect to e_j.

The correct notation is Pr(E_j > 0) = 0.5, where E_j is a random variable. Such misuse of notation shows his lack of basic knowledge in an area where he authoritatively propounds preposterous falsehood.--Palaeoviatalk 00:58, 5 August 2010 (UTC)

Palaeovia's concluding remarks

Anyone is welcome to propose improvement to the article. However, when someone with a history of writing utter nonsense proposes to re-introduce into this article his pet theory, for which after lengthy discussions I was not shown either a proof or a credible source, I am apt to be fervent in my pursuit of truth (mathematical and statistical, theoretical and empirical). My impression has been strenghthened that his understanding of the issue is superficial, his mathematical training is inadequate, and his interpretation is either "original research", or gross distortion of more carefully phrased, qualified statements from scholars.

Mathematicians generally promptly admit their errors, when pointed out, and proceed to seek the truth. Crackpots never admit their patent errors (usually they cannot understand mathematical and logical reasoning), and proceed to defend their pet theories to their last breath. I respect the former, and expose the latter.

In mathematics, truth is remarkably uncontroversial. It is not a matter of compromise. Of course what truth belongs to this article is a matter of debate and compromise. Excluding error and fallacy is my sole objective. I am open to be proved an idiot. I will learn from my errors, and improve. --Palaeoviatalk 23:02, 1 August 2010 (UTC)

Curious deletion of Palaeovia's posts on Crackpots and Sciolists

Given the persistent effort by Melcombe to delete my following post from this Talk page, I will explain its relevance:

My post on Crackpots and Sciolists

On the matter of mathematicians' honesty in facing up to their errors, the following example (from the article Andrew Wiles) of Andrew Wiles is exemplary:

The proof of Fermat's Last Theorem
Starting in the summer of 1986, based on successive progress of the previous few years of Gerhard Frey, Jean-Pierre Serre and Ken Ribet, Wiles realised that a proof of a limited form of the modularity theorem might then be in reach. He dedicated all of his research time to this problem in relative secrecy. In 1993, he presented his proof to the public for the first time at a conference in Cambridge. In August 1993, however, it turned out that the proof contained a gap. In desperation, Andrew Wiles tried to fill in this gap, but found out that the error he had made was a very fundamental one. According to Wiles, the crucial idea for circumventing, rather than closing this gap, came to him on 19 September 1994. Together with his former student Richard Taylor, he published a second paper which circumvented the gap and thus completed the proof. Both papers were published in 1995 in a special volume of the Annals of Mathematics.

How do you tell a mathematician from a mathematical crackpot?

• A mathematician occasionally makes subtle mistakes, understands his mistakes, and readily admits to them.
• A crackpot frequently makes obvious, elementary mistakes, cannot understand that they are mistakes, and never admits to any mistake.
• A mathematician confesses ignorance in fields beyond his expertise.
• A crackpot (a sciolist) propounds authoritatively on subjects of which he has but superficial and faulty knowledge.

Against sciolism:

• A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: there shallow draughts intoxicate the brain, and drinking largely sobers us again."

Explanation

The recent exchange concerns the truth of certain claims and the behavior of their author, upon being shown their falsity. An appropriate description for such behavior is "crackpot", a term not necessarily familiar to everyone. Giiven that I have had some experience in dealing with such behavior, and have given some serious thought to the same, my post above should help to clarify the precise sense in which I have employed the term, and serve to distinguish mathematicians from crackpots, in general.

I wonder at Melcombe's standard in policing this Talk page. I could point to certain other posts that, if such standard had been consistently applied, should have suffered at his hand.

Could my post have been singled out as a result of our previous exchange?--Palaeoviatalk 09:25, 10 August 2010 (UTC)

None of the above diatribe has anything to do with discussing what should be in the article and so should be deleted. Standards for talk pages are at WP:Talk. Melcombe (talk) 11:13, 10 August 2010 (UTC)

Express your views, by all means. Let each be his/her own judge. Deleting others' posts in a Talk page (meant for talks) seems devious. --Palaeoviatalk 11:33, 10 August 2010 (UTC)

If those posts breach our talk page guidelines, they can certainly be removed. Editors who make personal attacks against other editors can also be blocked or banned from editing Wikipedia. Comment on the content, not the contributor. --Avenue (talk) 13:00, 10 August 2010 (UTC)

Notation needs definition.

A casual reader pointed out here that there is no definition for most of the notation in the regression section. I don't know wikipedia standards for this, but E[], Var[], Cov[], \hat, epsilon, and pipe = conditional on all either need to be defined, eliminated, or a noted as defined on another page. It seems in the least squares page that they are defined with a link when first introduced. — Preceding unsigned comment added by 108.75.137.21 (talk) 14:47, 22 June 2011 (UTC)

Cross-Cultural Differences in recognizing and adjusting to a regression toward the mean.

A recent study performed by Roy Spina et al. found that there are cultural differences in being able to account for the regression toward the mean, and I think that this may be found in other studies and would add to this article. Here is the citation for his article. Spina, R. R., Ji, L., Ross, M., Li, Y., & Zhang, Z. (2010). Why best cannot last: Cultural differences in predicting regression toward the mean. Asian Journal Of Social Psychology, 13(3), 153-162. doi:10.1111/j.1467-839X.2010.01310.x Fotherge (talk) 21:19, 7 February 2012 (UTC)

Please select non-controversial examples illustrating regression to mean.

Have removed a reference to alleged criticism of UK speed cameras partly because it appeared to argue an unrelated point about speed cameras being an unproductive use of road safety funds. A good illustrative example of regression to the mean should be clear & easy to understand and should not drag in any secondary issues which could detract from the idea being explained.

Noel darlow (talk) 21:46, 23 October 2013 (UTC)

Reinserted with offending sentence removed and two additional references. Qwfp (talk) 19:30, 24 October 2013 (UTC)
Great :) It does make a good, topical example of R2M when it's worded to be camera-neutral. Noel darlow (talk) 01:07, 25 October 2013 (UTC)