Why isn' t high order polynomial a good fit?
$begingroup$
Let's say I have a set of data points $(x_i,y_i), i=1,2,...,N$, and I want to approximate it using a polynomial $p(x)=sum_{i=0}^n a_i x^i$ with a least squares fit (so $n<N$).
I know that the coefficient $R^2$ is a measure for goodness of fit. But as I increase the order $n$ of the polynomial $p$, $R^2$ approaches $1$. (In the extreme case when $n=N$ the fit is an interpolation with $R^2=1$.)
But a high order polynomial fit is likely an unphysical result and doesn't describe the population well. So is there a mathematical characteristic or coefficient or model that describes and explains this?
statistics statistical-inference
asked Feb 10 '15 at 11:49
MinethlosMinethlos
9561127
$endgroup$
add a comment |
$begingroup$
Let's say I have a set of data points $(x_i,y_i), i=1,2,...,N$, and I want to approximate it using a polynomial $p(x)=sum_{i=0}^n a_i x^i$ with a least squares fit (so $n<N$).
I know that the coefficient $R^2$ is a measure for goodness of fit. But as I increase the order $n$ of the polynomial $p$, $R^2$ approaches $1$. (In the extreme case when $n=N$ the fit is an interpolation with $R^2=1$.)
But a high order polynomial fit is likely an unphysical result and doesn't describe the population well. So is there a mathematical characteristic or coefficient or model that describes and explains this?
statistics statistical-inference
asked Feb 10 '15 at 11:49
MinethlosMinethlos
9561127
$endgroup$
4
$begingroup$
en.wikipedia.org/wiki/Overfitting
$endgroup$
– M.B.
Feb 10 '15 at 11:53
$begingroup$
"a high order polynomial fit is likely an unphysical result and doesn't describe the population well": I don't agree, every smooth function is well approximated by its Taylor polynomial. For instance, it works very well for an exponential growth.
$endgroup$
– Yves Daoust
Feb 16 '15 at 10:37
add a comment |
$begingroup$
Let's say I have a set of data points $(x_i,y_i), i=1,2,...,N$, and I want to approximate it using a polynomial $p(x)=sum_{i=0}^n a_i x^i$ with a least squares fit (so $n<N$).
I know that the coefficient $R^2$ is a measure for goodness of fit. But as I increase the order $n$ of the polynomial $p$, $R^2$ approaches $1$. (In the extreme case when $n=N$ the fit is an interpolation with $R^2=1$.)
But a high order polynomial fit is likely an unphysical result and doesn't describe the population well. So is there a mathematical characteristic or coefficient or model that describes and explains this?
statistics statistical-inference
asked Feb 10 '15 at 11:49
MinethlosMinethlos
9561127
$endgroup$
Let's say I have a set of data points $(x_i,y_i), i=1,2,...,N$, and I want to approximate it using a polynomial $p(x)=sum_{i=0}^n a_i x^i$ with a least squares fit (so $n<N$).
I know that the coefficient $R^2$ is a measure for goodness of fit. But as I increase the order $n$ of the polynomial $p$, $R^2$ approaches $1$. (In the extreme case when $n=N$ the fit is an interpolation with $R^2=1$.)
But a high order polynomial fit is likely an unphysical result and doesn't describe the population well. So is there a mathematical characteristic or coefficient or model that describes and explains this?
statistics statistical-inference
statistics statistical-inference
asked Feb 10 '15 at 11:49
MinethlosMinethlos
9561127
asked Feb 10 '15 at 11:49
MinethlosMinethlos
9561127
edited Feb 16 '15 at 10:26
Minethlos
asked Feb 10 '15 at 11:49
MinethlosMinethlos
9561127
asked Feb 10 '15 at 11:49
MinethlosMinethlos
9561127
asked Feb 10 '15 at 11:49
MinethlosMinethlos
9561127
9561127
4
$begingroup$
en.wikipedia.org/wiki/Overfitting
$endgroup$
– M.B.
Feb 10 '15 at 11:53
$begingroup$
"a high order polynomial fit is likely an unphysical result and doesn't describe the population well": I don't agree, every smooth function is well approximated by its Taylor polynomial. For instance, it works very well for an exponential growth.
$endgroup$
– Yves Daoust
Feb 16 '15 at 10:37
add a comment |
4
$begingroup$
en.wikipedia.org/wiki/Overfitting
$endgroup$
– M.B.
Feb 10 '15 at 11:53
$begingroup$
"a high order polynomial fit is likely an unphysical result and doesn't describe the population well": I don't agree, every smooth function is well approximated by its Taylor polynomial. For instance, it works very well for an exponential growth.
$endgroup$
– Yves Daoust
Feb 16 '15 at 10:37
4
4
$begingroup$
en.wikipedia.org/wiki/Overfitting
$endgroup$
– M.B.
Feb 10 '15 at 11:53
$begingroup$
en.wikipedia.org/wiki/Overfitting
$endgroup$
– M.B.
Feb 10 '15 at 11:53
$begingroup$
"a high order polynomial fit is likely an unphysical result and doesn't describe the population well": I don't agree, every smooth function is well approximated by its Taylor polynomial. For instance, it works very well for an exponential growth.
$endgroup$
– Yves Daoust
Feb 16 '15 at 10:37
$begingroup$
"a high order polynomial fit is likely an unphysical result and doesn't describe the population well": I don't agree, every smooth function is well approximated by its Taylor polynomial. For instance, it works very well for an exponential growth.
$endgroup$
– Yves Daoust
Feb 16 '15 at 10:37
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
I think what you might be looking for is the Akaike information criterion. It takes the number of parameters and the maximised value of the liklihood function to produce a number; the AIC says the model which produces the minimum value is preferred. Essentially it rewards well-fitting but penalises use of extra parameters.
http://en.m.wikipedia.org/wiki/Akaike_information_criterion
answered Feb 16 '15 at 10:45
FireGardenFireGarden
2,84421525
$endgroup$
$begingroup$
Yes, the information theory background of AIC is enlightening. Thank you.
$endgroup$
– Minethlos
Feb 16 '15 at 11:32
add a comment |
$begingroup$
Why you use R^2 for the goodness of fit, why not just use Mean Square Error or Root Mean Square Error? Higher order polynomials are prone to local fluctuations, what you can do is to use regularization to reduce the artifacts, e.g., you can add a penalty term to address the second order derivation of the high order polynomial.
answered Dec 18 '18 at 2:47
QZHuaQZHua
112
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f1141956%2fwhy-isn-t-high-order-polynomial-a-good-fit%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
I think what you might be looking for is the Akaike information criterion. It takes the number of parameters and the maximised value of the liklihood function to produce a number; the AIC says the model which produces the minimum value is preferred. Essentially it rewards well-fitting but penalises use of extra parameters.
http://en.m.wikipedia.org/wiki/Akaike_information_criterion
answered Feb 16 '15 at 10:45
FireGardenFireGarden
2,84421525
$endgroup$
$begingroup$
Yes, the information theory background of AIC is enlightening. Thank you.
$endgroup$
– Minethlos
Feb 16 '15 at 11:32
add a comment |
$begingroup$
I think what you might be looking for is the Akaike information criterion. It takes the number of parameters and the maximised value of the liklihood function to produce a number; the AIC says the model which produces the minimum value is preferred. Essentially it rewards well-fitting but penalises use of extra parameters.
http://en.m.wikipedia.org/wiki/Akaike_information_criterion
answered Feb 16 '15 at 10:45
FireGardenFireGarden
2,84421525
$endgroup$
$begingroup$
Yes, the information theory background of AIC is enlightening. Thank you.
$endgroup$
– Minethlos
Feb 16 '15 at 11:32
add a comment |
$begingroup$
I think what you might be looking for is the Akaike information criterion. It takes the number of parameters and the maximised value of the liklihood function to produce a number; the AIC says the model which produces the minimum value is preferred. Essentially it rewards well-fitting but penalises use of extra parameters.
http://en.m.wikipedia.org/wiki/Akaike_information_criterion
answered Feb 16 '15 at 10:45
FireGardenFireGarden
2,84421525
$endgroup$
I think what you might be looking for is the Akaike information criterion. It takes the number of parameters and the maximised value of the liklihood function to produce a number; the AIC says the model which produces the minimum value is preferred. Essentially it rewards well-fitting but penalises use of extra parameters.
http://en.m.wikipedia.org/wiki/Akaike_information_criterion
answered Feb 16 '15 at 10:45
FireGardenFireGarden
2,84421525
answered Feb 16 '15 at 10:45
FireGardenFireGarden
2,84421525
answered Feb 16 '15 at 10:45
FireGardenFireGarden
2,84421525
answered Feb 16 '15 at 10:45
FireGardenFireGarden
2,84421525
2,84421525
$begingroup$
Yes, the information theory background of AIC is enlightening. Thank you.
$endgroup$
– Minethlos
Feb 16 '15 at 11:32
add a comment |
$begingroup$
Yes, the information theory background of AIC is enlightening. Thank you.
$endgroup$
– Minethlos
Feb 16 '15 at 11:32
$begingroup$
Yes, the information theory background of AIC is enlightening. Thank you.
$endgroup$
– Minethlos
Feb 16 '15 at 11:32
$begingroup$
Yes, the information theory background of AIC is enlightening. Thank you.
$endgroup$
– Minethlos
Feb 16 '15 at 11:32
add a comment |
$begingroup$
Why you use R^2 for the goodness of fit, why not just use Mean Square Error or Root Mean Square Error? Higher order polynomials are prone to local fluctuations, what you can do is to use regularization to reduce the artifacts, e.g., you can add a penalty term to address the second order derivation of the high order polynomial.
answered Dec 18 '18 at 2:47
QZHuaQZHua
112
$endgroup$
add a comment |
$begingroup$
Why you use R^2 for the goodness of fit, why not just use Mean Square Error or Root Mean Square Error? Higher order polynomials are prone to local fluctuations, what you can do is to use regularization to reduce the artifacts, e.g., you can add a penalty term to address the second order derivation of the high order polynomial.
answered Dec 18 '18 at 2:47
QZHuaQZHua
112
$endgroup$
add a comment |
$begingroup$
Why you use R^2 for the goodness of fit, why not just use Mean Square Error or Root Mean Square Error? Higher order polynomials are prone to local fluctuations, what you can do is to use regularization to reduce the artifacts, e.g., you can add a penalty term to address the second order derivation of the high order polynomial.
answered Dec 18 '18 at 2:47
QZHuaQZHua
112
$endgroup$
Why you use R^2 for the goodness of fit, why not just use Mean Square Error or Root Mean Square Error? Higher order polynomials are prone to local fluctuations, what you can do is to use regularization to reduce the artifacts, e.g., you can add a penalty term to address the second order derivation of the high order polynomial.
answered Dec 18 '18 at 2:47
QZHuaQZHua
112
answered Dec 18 '18 at 2:47
QZHuaQZHua
112
answered Dec 18 '18 at 2:47
QZHuaQZHua
112
answered Dec 18 '18 at 2:47
QZHuaQZHua
112
112
add a comment |
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f1141956%2fwhy-isn-t-high-order-polynomial-a-good-fit%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
4
$begingroup$
en.wikipedia.org/wiki/Overfitting
$endgroup$
– M.B.
Feb 10 '15 at 11:53
$begingroup$
"a high order polynomial fit is likely an unphysical result and doesn't describe the population well": I don't agree, every smooth function is well approximated by its Taylor polynomial. For instance, it works very well for an exponential growth.
$endgroup$
– Yves Daoust
Feb 16 '15 at 10:37