Talk:Simple linear regression
|WikiProject Statistics||(Rated C-class, Mid-importance)|
|WikiProject Mathematics||(Rated C-class, Mid-importance)|
- 1 edits
- 2 Regression articles discussion July 2009
- 3 Linear regression assumptions
- 4 Broken link to sample correlation
- 5 carets
- 6 Numerical Example
- 7 Broken link to Total least squares
- 8 By using calculus
- 9 Fitting the regression line
- 10 Notation can lead to mistakes
- 11 beta hat
- 12 why is R^2 = r^2 ?
I edited the page for math content (although I'm no expert on latex, either), added a section on inference and a numerical example. I don't know if the numerical example is helpful. EconProf86 22:04, 31 July 2007 (UTC)
Regression articles discussion July 2009
A discussion of content overlap of some regression-related articles has been started at Talk:Linear least squares#Merger proposal but it isn't really just a question of merging and no actual merge proposal has been made. Melcombe (talk) 11:45, 14 July 2009 (UTC)
Linear regression assumptions
Perhaps there should be a straight-forward section on assumptions, and how to check them, like this:
* 1) That there is a linear relationship between independent and dependent variable(s). How to Check: Make an XY scatter plot, then look for data grouping along a line, instead of along a curve. * 2) That the data are homoskedastic, meaning that errors do not tend to get bigger (or smaller), as a trend, as independent variables change. How to Check: Make a residual plot, then see if it is symmetric, or make an XY scatter plot, and see if the points do not tend to spread as they progress toward the left, or toward the right. If the scatter plot points look like they get farther apart as they go from left to right (or vice versa), then the data are not homoskedastic. * 3) That the data are normally distributed, which would meet the three following conditions: * a) Unimodal_function: How to Check: Make a histogram of the data, then look for only one major peak, instead of many. * b) Symmetric, or Unskewed Data Distribution: How to Check Skewness: Make that same histogram, then compare the left and right tails - Do they look to be the same size? Or is the graph 'leaning' one way or another? * c) Kurtosis is approximately Zero: How to Check: Make that same histogram, then compare its peakedness to a normal distribution. Is it 'peakier', or less 'peaky'? Are the data points more clustered than a normal distribution?
1) is false. That's not what 'linear' means. The linearity refers to a linear relationship between y and the PARAMETERS, i.e. the alpha and the betas. A quadratic relationship between y and x is still (a little paradoxically) a 'linear regression'. In mathematics an equation is said to be linear or nonlinear if it's linear or nonlinear in the UNKNOWNS, and in the regression setting it's the alpha and betas that are unknown.
The link Correlation#Sample_correlation in the first section is broken (there's no such anchor in the page).
Where should this point to? Correlation#Pearson.27s_product-moment_coefficient ?
Can anyone more mathematically literate tell me if there's a reason the α and β subscripts of standard error s have carets on top of them in the section "normality assumption", but not in the numerical example? - Kyle at 6:35pm CST, 4 April 2011 —Preceding unsigned comment added by 18.104.22.168 (talk) 23:38, 5 April 2011 (UTC)
There is an unfortunate basic error in the data. If plotted, an odd cadence in the x-positions can be noticed. This is because the original data were in inches, and the heights have been converted to metres, with rounding to the nearest centimetre. This is wrong, and has visible effects. Also, the line fit parameters change.
Slope Const. 61.2722 -39.062 wrongly rounding to the nearest centimetre. 61.6746 -39.7468 conversion without rounding.
And naturally, all the confidence intervals, etc. change as well. The fix is simple enough. Replace the original x by round(x/0.0254)*0.0254. I had reported this problem in 2008, but in the multiple reorganisations of the linear regression article, this was lost. There is not much point in discussing fancy calculations if the data are corrupt.
Later, it occurred to me to consider whether the weights might have been given in pounds. The results were odd in another way. Using the conversion 1KG = 2.20462234 lbs used in the USA, the weights are
115.1033 117.1095 120.1078 123.1061 126.1044 129.1247 132.123 135.1213 139.1337 142.132 146.1224 150.1348 154.1472 159.1517 164.1562 114.862 116.864 119.856 122.848 125.84 128.854 131.846 134.838 138.842 141.834 145.816 149.82 153.824 158.818 163.812
By using calculus
It is not obvious to me how to solve that equation to derive the forms below it in the "Fitting the regression line" section. The text says "By using either calculus, the geometry of inner product spaces or simply expanding to get a quadratic in α and β, it can be shown that the values of α and β that minimize the objective function Q are..."
How do I use these methods to show that? I'd like to see this expanded. Spell out the steps in the derivation more explicitly. Perhaps I could figure out how to do it using the "geometry of inner product spaces" method if I read the linked article. To solve with calculus, I would differentiate by α and β and find where the derivative was equal to zero and the second derivative was positive. I... forgot how to do this with two variables, and I especially don't know how to do this with that summation.
Ideally, I'd like to see a link to an article that describes these methods (like the "geometry of inner product spaces" method), and also at least some of the intermediate steps in the derivation. Gnebulon (talk) 02:57, 9 November 2011 (UTC)
- It's a matter of ploughing ahead. The general idea begins with minimising
so the plan is to minimise E by choice of in
The first step is to expand the contents of the summation, thus There are endless variations on this, with , a,b, (or vice-versa) and m,c.
Then further expansion,
Now apply the rules of the calculus, with a separate differentiation for each of the parameters. As usual, the extremum is to be found where the slope has the value zero. I prefer to avoid the horizontal bar in dy/dx as it is not a normal sort of fraction, so you shouldn't cancel the ds for example. But that's just me.
The twos can be factored out, so
Remembering that a rearrangement gives
Which is to say that the line (of whatever slope) goes through the average point because
Notice that the second differential is constant, 2: a positive number. So this extremum is a minimum.
Minimising with respect to is the first half. The second is to minimise with respect to and now it becomes clear why collecting the terms for E would have been a waste of effort.
As before, the twos can be factored out, so
The second differential for this is which must be positive, and so this extremum is also a minimum.
Remembering that and substituting for
Multiplying out and re-arranging,
Multiplying top and bottom by N renders this less typographically formidable. Other variations are possible via the use of and as appropriate.
Fitting the regression line
In ==Fitting the regression line== shouldn't the expression immediately before the one containing Cov & Var have 1/n as the multiplier before the - x.y and before the - x^2 terms? This is the form of the expression which is often used in computing to generate a straight-line fit to set of "bumpy" data. As expressed here it does not work and moreover does not follow mathematically from the preceding expression! However with the 1/n terms in place it appears to produce the correct result. 1/n is unique to these two terms ONLY and therefore does NOT cancel?!? But then I am a Physicist and not a Mathematician so I may have missed something??? Chris B. 6:45pm PST on 24th. Mar. 2013. — Preceding unsigned comment added by 22.214.171.124 (talk) 00:48, 25 March 2013 (UTC)
- Have a look at the section below "By using calculus" which steps through a derivation and also mentions multiplying top and bottom by N. Incidentally, I also studied Physics. NickyMcLean (talk) 02:24, 25 March 2013 (UTC)
Notation can lead to mistakes
The expression can lead to mistakes if you use the sample variance instead of variance. Since every spreadsheet gives you the sample variance it is likely that people can use this formula incorrecly (as one of my students just did in one assignment). It should be better to stress that Var is not the sample variance. — Preceding unsigned comment added by 25pietro (talk • contribs) 07:11, 13 June 2014 (UTC)
I was confused by the formula for , I wonder if the second one should have more parentheses around sums, like (as product has precedence over summation) - but I am no mathematician nor English, so perhaps do not know conventions. Can someone look at it and possibly fix it? I know it is logical to at least assume the parentheses, but this is introductory and should be as precise as possible.Drabek (talk) 19:08, 21 March 2014 (UTC)
why is R^2 = r^2 ?
It would be nice to include a section proving why R^2 = r^2 in this case of Simple_linear_regression.
Wikipedia page on R^2 only says:
Similarly, in linear least squares regression with an estimated intercept term, R2 equals the square of the Pearson correlation coefficient between the observed and modeled (predicted) data values of the dependent variable.
i.e., R^2 = r(Y, Yhat)^2, which is proved in the Wikipedia page on Pearson_product-moment_correlation_coefficient (Section 5)