Smart way to compute Residual Sum of Squares (RSS) in Multiple Linear Regression
$begingroup$
Is there any smarter way to compute Residual Sum of Squares(RSS) in Multiple Linear Regression other then fitting the model -> find coefficients -> find fitted values -> find residuals -> find norm of residuals... If I need only RSS and nothing else. For example, in best subset selection, we need to determine RSS of many reduced models..
linear-algebra regression
edited Feb 26 '18 at 14:37
Did
248k23225463
asked Apr 26 '13 at 18:41
Oleg ShirokikhOleg Shirokikh
1334
$endgroup$
add a comment |
$begingroup$
Is there any smarter way to compute Residual Sum of Squares(RSS) in Multiple Linear Regression other then fitting the model -> find coefficients -> find fitted values -> find residuals -> find norm of residuals... If I need only RSS and nothing else. For example, in best subset selection, we need to determine RSS of many reduced models..
linear-algebra regression
edited Feb 26 '18 at 14:37
Did
248k23225463
asked Apr 26 '13 at 18:41
Oleg ShirokikhOleg Shirokikh
1334
$endgroup$
add a comment |
$begingroup$
Is there any smarter way to compute Residual Sum of Squares(RSS) in Multiple Linear Regression other then fitting the model -> find coefficients -> find fitted values -> find residuals -> find norm of residuals... If I need only RSS and nothing else. For example, in best subset selection, we need to determine RSS of many reduced models..
linear-algebra regression
edited Feb 26 '18 at 14:37
Did
248k23225463
asked Apr 26 '13 at 18:41
Oleg ShirokikhOleg Shirokikh
1334
$endgroup$
Is there any smarter way to compute Residual Sum of Squares(RSS) in Multiple Linear Regression other then fitting the model -> find coefficients -> find fitted values -> find residuals -> find norm of residuals... If I need only RSS and nothing else. For example, in best subset selection, we need to determine RSS of many reduced models..
linear-algebra regression
linear-algebra regression
edited Feb 26 '18 at 14:37
Did
248k23225463
asked Apr 26 '13 at 18:41
Oleg ShirokikhOleg Shirokikh
1334
edited Feb 26 '18 at 14:37
Did
248k23225463
asked Apr 26 '13 at 18:41
Oleg ShirokikhOleg Shirokikh
1334
edited Feb 26 '18 at 14:37
Did
248k23225463
edited Feb 26 '18 at 14:37
Did
248k23225463
edited Feb 26 '18 at 14:37
Did
248k23225463
248k23225463
asked Apr 26 '13 at 18:41
Oleg ShirokikhOleg Shirokikh
1334
asked Apr 26 '13 at 18:41
Oleg ShirokikhOleg Shirokikh
1334
asked Apr 26 '13 at 18:41
Oleg ShirokikhOleg Shirokikh
1334
1334
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
[update] Upps, just saw your comment at @Nameless, that you have 10000 variables. So I think, that covariaton-approach is useless too. Should I delete the answer? [/update]
One method needs only the inversion of the covariation matrix, don't know whether this is already smart enough?
Construct the datamatrix $D$ with the top row from the rowvector of $Y$-values, then the rowvectors of $X$-variables/values. If $X$ and $Y$-variables are not centered append one more row containing only 1. (If you have, say 3 $X$-variables and $n$ cases, you have then a $4 times n$ or $ 5 times n$ matrix).
Then compute the dotproduct of D with itself $C= D cdot D^t$ and the inverse $B=C^{-1}$ Then take the reciprocal of the top-left entry of $B$, say $s = 1/B_{1,1}$ Then s is the sum-of-squares of the residuals.
answered Apr 26 '13 at 22:11
Gottfried HelmsGottfried Helms
23.5k24599
$endgroup$
add a comment |
$begingroup$
If we write in matrix form $$Y = Xbeta + epsilon$$ where
- $Y, epsilon$ are $N times 1$
- $X$ is $N times k$
- $beta$ is $k times 1$
The RSS is defined as $$text{RSS} = frac{1}{N} Vert Y - hat{Y} Vert^2 $$
where $$hat{Y} = X hat{beta}$$
Let's say you've used OLS as the method to estimate your regressors, i.e.
$$hat{beta} = (X^T X)^{-1}X^T Y$$This means that
$$hat{Y} = X (X^T X)^{-1}X^T Y = P_XY$$where $P_X=X (X^T X)^{-1}X^T$is a projector matrix onto the span of the columns of $X$. Replacing this expression in the RSS gives
$$text{RSS} = frac{1}{N} Vert Y - hat{Y} Vert^2 = frac{1}{N} Vert Y - P_XY Vert^2 = frac{1}{N} Vert (I - P_X)Y Vert^2 = frac{1}{N} Vert P_X^{perp}Y Vert^2$$
where $P_X^{perp} = I - P_X$ in the projector onto the null space of the space spanned by the columns of $X$. We can write
$$RSS = frac{1}{N} Y^TP_X^{perp}P_X^{perp}Y = frac{1}{N} Y^TP_X^{perp}Y$$
Hence
$$RSS = frac{1}{N} Y^TP_X^{perp}Y$$
where
$$P_X^{perp} = I - X(X^TX)^{-1}X^T$$
which is true since $P_X^{perp}$ is idompontent.
So the above equation suggests, that given $Y,X$ (which you already have), compute the RSS.
answered Sep 2 '18 at 16:10
Ahmad BazziAhmad Bazzi
8,3622824
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f373771%2fsmart-way-to-compute-residual-sum-of-squares-rss-in-multiple-linear-regression%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
[update] Upps, just saw your comment at @Nameless, that you have 10000 variables. So I think, that covariaton-approach is useless too. Should I delete the answer? [/update]
One method needs only the inversion of the covariation matrix, don't know whether this is already smart enough?
Construct the datamatrix $D$ with the top row from the rowvector of $Y$-values, then the rowvectors of $X$-variables/values. If $X$ and $Y$-variables are not centered append one more row containing only 1. (If you have, say 3 $X$-variables and $n$ cases, you have then a $4 times n$ or $ 5 times n$ matrix).
Then compute the dotproduct of D with itself $C= D cdot D^t$ and the inverse $B=C^{-1}$ Then take the reciprocal of the top-left entry of $B$, say $s = 1/B_{1,1}$ Then s is the sum-of-squares of the residuals.
answered Apr 26 '13 at 22:11
Gottfried HelmsGottfried Helms
23.5k24599
$endgroup$
add a comment |
$begingroup$
[update] Upps, just saw your comment at @Nameless, that you have 10000 variables. So I think, that covariaton-approach is useless too. Should I delete the answer? [/update]
One method needs only the inversion of the covariation matrix, don't know whether this is already smart enough?
Construct the datamatrix $D$ with the top row from the rowvector of $Y$-values, then the rowvectors of $X$-variables/values. If $X$ and $Y$-variables are not centered append one more row containing only 1. (If you have, say 3 $X$-variables and $n$ cases, you have then a $4 times n$ or $ 5 times n$ matrix).
Then compute the dotproduct of D with itself $C= D cdot D^t$ and the inverse $B=C^{-1}$ Then take the reciprocal of the top-left entry of $B$, say $s = 1/B_{1,1}$ Then s is the sum-of-squares of the residuals.
answered Apr 26 '13 at 22:11
Gottfried HelmsGottfried Helms
23.5k24599
$endgroup$
add a comment |
$begingroup$
[update] Upps, just saw your comment at @Nameless, that you have 10000 variables. So I think, that covariaton-approach is useless too. Should I delete the answer? [/update]
One method needs only the inversion of the covariation matrix, don't know whether this is already smart enough?
Construct the datamatrix $D$ with the top row from the rowvector of $Y$-values, then the rowvectors of $X$-variables/values. If $X$ and $Y$-variables are not centered append one more row containing only 1. (If you have, say 3 $X$-variables and $n$ cases, you have then a $4 times n$ or $ 5 times n$ matrix).
Then compute the dotproduct of D with itself $C= D cdot D^t$ and the inverse $B=C^{-1}$ Then take the reciprocal of the top-left entry of $B$, say $s = 1/B_{1,1}$ Then s is the sum-of-squares of the residuals.
answered Apr 26 '13 at 22:11
Gottfried HelmsGottfried Helms
23.5k24599
$endgroup$
[update] Upps, just saw your comment at @Nameless, that you have 10000 variables. So I think, that covariaton-approach is useless too. Should I delete the answer? [/update]
One method needs only the inversion of the covariation matrix, don't know whether this is already smart enough?
Construct the datamatrix $D$ with the top row from the rowvector of $Y$-values, then the rowvectors of $X$-variables/values. If $X$ and $Y$-variables are not centered append one more row containing only 1. (If you have, say 3 $X$-variables and $n$ cases, you have then a $4 times n$ or $ 5 times n$ matrix).
Then compute the dotproduct of D with itself $C= D cdot D^t$ and the inverse $B=C^{-1}$ Then take the reciprocal of the top-left entry of $B$, say $s = 1/B_{1,1}$ Then s is the sum-of-squares of the residuals.
answered Apr 26 '13 at 22:11
Gottfried HelmsGottfried Helms
23.5k24599
edited Apr 26 '13 at 22:18
answered Apr 26 '13 at 22:11
Gottfried HelmsGottfried Helms
23.5k24599
answered Apr 26 '13 at 22:11
Gottfried HelmsGottfried Helms
23.5k24599
answered Apr 26 '13 at 22:11
Gottfried HelmsGottfried Helms
23.5k24599
23.5k24599
add a comment |
add a comment |
$begingroup$
If we write in matrix form $$Y = Xbeta + epsilon$$ where
- $Y, epsilon$ are $N times 1$
- $X$ is $N times k$
- $beta$ is $k times 1$
The RSS is defined as $$text{RSS} = frac{1}{N} Vert Y - hat{Y} Vert^2 $$
where $$hat{Y} = X hat{beta}$$
Let's say you've used OLS as the method to estimate your regressors, i.e.
$$hat{beta} = (X^T X)^{-1}X^T Y$$This means that
$$hat{Y} = X (X^T X)^{-1}X^T Y = P_XY$$where $P_X=X (X^T X)^{-1}X^T$is a projector matrix onto the span of the columns of $X$. Replacing this expression in the RSS gives
$$text{RSS} = frac{1}{N} Vert Y - hat{Y} Vert^2 = frac{1}{N} Vert Y - P_XY Vert^2 = frac{1}{N} Vert (I - P_X)Y Vert^2 = frac{1}{N} Vert P_X^{perp}Y Vert^2$$
where $P_X^{perp} = I - P_X$ in the projector onto the null space of the space spanned by the columns of $X$. We can write
$$RSS = frac{1}{N} Y^TP_X^{perp}P_X^{perp}Y = frac{1}{N} Y^TP_X^{perp}Y$$
Hence
$$RSS = frac{1}{N} Y^TP_X^{perp}Y$$
where
$$P_X^{perp} = I - X(X^TX)^{-1}X^T$$
which is true since $P_X^{perp}$ is idompontent.
So the above equation suggests, that given $Y,X$ (which you already have), compute the RSS.
answered Sep 2 '18 at 16:10
Ahmad BazziAhmad Bazzi
8,3622824
$endgroup$
add a comment |
$begingroup$
If we write in matrix form $$Y = Xbeta + epsilon$$ where
- $Y, epsilon$ are $N times 1$
- $X$ is $N times k$
- $beta$ is $k times 1$
The RSS is defined as $$text{RSS} = frac{1}{N} Vert Y - hat{Y} Vert^2 $$
where $$hat{Y} = X hat{beta}$$
Let's say you've used OLS as the method to estimate your regressors, i.e.
$$hat{beta} = (X^T X)^{-1}X^T Y$$This means that
$$hat{Y} = X (X^T X)^{-1}X^T Y = P_XY$$where $P_X=X (X^T X)^{-1}X^T$is a projector matrix onto the span of the columns of $X$. Replacing this expression in the RSS gives
$$text{RSS} = frac{1}{N} Vert Y - hat{Y} Vert^2 = frac{1}{N} Vert Y - P_XY Vert^2 = frac{1}{N} Vert (I - P_X)Y Vert^2 = frac{1}{N} Vert P_X^{perp}Y Vert^2$$
where $P_X^{perp} = I - P_X$ in the projector onto the null space of the space spanned by the columns of $X$. We can write
$$RSS = frac{1}{N} Y^TP_X^{perp}P_X^{perp}Y = frac{1}{N} Y^TP_X^{perp}Y$$
Hence
$$RSS = frac{1}{N} Y^TP_X^{perp}Y$$
where
$$P_X^{perp} = I - X(X^TX)^{-1}X^T$$
which is true since $P_X^{perp}$ is idompontent.
So the above equation suggests, that given $Y,X$ (which you already have), compute the RSS.
answered Sep 2 '18 at 16:10
Ahmad BazziAhmad Bazzi
8,3622824
$endgroup$
add a comment |
$begingroup$
If we write in matrix form $$Y = Xbeta + epsilon$$ where
- $Y, epsilon$ are $N times 1$
- $X$ is $N times k$
- $beta$ is $k times 1$
The RSS is defined as $$text{RSS} = frac{1}{N} Vert Y - hat{Y} Vert^2 $$
where $$hat{Y} = X hat{beta}$$
Let's say you've used OLS as the method to estimate your regressors, i.e.
$$hat{beta} = (X^T X)^{-1}X^T Y$$This means that
$$hat{Y} = X (X^T X)^{-1}X^T Y = P_XY$$where $P_X=X (X^T X)^{-1}X^T$is a projector matrix onto the span of the columns of $X$. Replacing this expression in the RSS gives
$$text{RSS} = frac{1}{N} Vert Y - hat{Y} Vert^2 = frac{1}{N} Vert Y - P_XY Vert^2 = frac{1}{N} Vert (I - P_X)Y Vert^2 = frac{1}{N} Vert P_X^{perp}Y Vert^2$$
where $P_X^{perp} = I - P_X$ in the projector onto the null space of the space spanned by the columns of $X$. We can write
$$RSS = frac{1}{N} Y^TP_X^{perp}P_X^{perp}Y = frac{1}{N} Y^TP_X^{perp}Y$$
Hence
$$RSS = frac{1}{N} Y^TP_X^{perp}Y$$
where
$$P_X^{perp} = I - X(X^TX)^{-1}X^T$$
which is true since $P_X^{perp}$ is idompontent.
So the above equation suggests, that given $Y,X$ (which you already have), compute the RSS.
answered Sep 2 '18 at 16:10
Ahmad BazziAhmad Bazzi
8,3622824
$endgroup$
If we write in matrix form $$Y = Xbeta + epsilon$$ where
- $Y, epsilon$ are $N times 1$
- $X$ is $N times k$
- $beta$ is $k times 1$
The RSS is defined as $$text{RSS} = frac{1}{N} Vert Y - hat{Y} Vert^2 $$
where $$hat{Y} = X hat{beta}$$
Let's say you've used OLS as the method to estimate your regressors, i.e.
$$hat{beta} = (X^T X)^{-1}X^T Y$$This means that
$$hat{Y} = X (X^T X)^{-1}X^T Y = P_XY$$where $P_X=X (X^T X)^{-1}X^T$is a projector matrix onto the span of the columns of $X$. Replacing this expression in the RSS gives
$$text{RSS} = frac{1}{N} Vert Y - hat{Y} Vert^2 = frac{1}{N} Vert Y - P_XY Vert^2 = frac{1}{N} Vert (I - P_X)Y Vert^2 = frac{1}{N} Vert P_X^{perp}Y Vert^2$$
where $P_X^{perp} = I - P_X$ in the projector onto the null space of the space spanned by the columns of $X$. We can write
$$RSS = frac{1}{N} Y^TP_X^{perp}P_X^{perp}Y = frac{1}{N} Y^TP_X^{perp}Y$$
Hence
$$RSS = frac{1}{N} Y^TP_X^{perp}Y$$
where
$$P_X^{perp} = I - X(X^TX)^{-1}X^T$$
which is true since $P_X^{perp}$ is idompontent.
So the above equation suggests, that given $Y,X$ (which you already have), compute the RSS.
answered Sep 2 '18 at 16:10
Ahmad BazziAhmad Bazzi
8,3622824
answered Sep 2 '18 at 16:10
Ahmad BazziAhmad Bazzi
8,3622824
answered Sep 2 '18 at 16:10
Ahmad BazziAhmad Bazzi
8,3622824
answered Sep 2 '18 at 16:10
Ahmad BazziAhmad Bazzi
8,3622824
8,3622824
add a comment |
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f373771%2fsmart-way-to-compute-residual-sum-of-squares-rss-in-multiple-linear-regression%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
