Functional Gradient Descent and Functional Taylor Expansion
$begingroup$
The questions are based on the below screenshots.
Can somebody explain how the functional Taylor expansion is related to a "standard" function Taylor expansion? In particular, I am concerned with this term
$$
C(F+ epsilon f) = C(F) + epsilon <nabla C(F), f>
$$
where $<cdot{}, cdot{}>$ is some suitable inner product.Why is it in general not possible to choose $f = - nabla C(F)$?
Source: Functional Gradient Descent for combining hypotheses by Mason et al. (1999)
functional-analysis taylor-expansion gradient-descent
edited Dec 5 '18 at 9:22
José Carlos Santos
157k22126227
asked Dec 5 '18 at 9:15
rk92rk92
82
$endgroup$
add a comment |
$begingroup$
The questions are based on the below screenshots.
Can somebody explain how the functional Taylor expansion is related to a "standard" function Taylor expansion? In particular, I am concerned with this term
$$
C(F+ epsilon f) = C(F) + epsilon <nabla C(F), f>
$$
where $<cdot{}, cdot{}>$ is some suitable inner product.Why is it in general not possible to choose $f = - nabla C(F)$?
Source: Functional Gradient Descent for combining hypotheses by Mason et al. (1999)
functional-analysis taylor-expansion gradient-descent
edited Dec 5 '18 at 9:22
José Carlos Santos
157k22126227
asked Dec 5 '18 at 9:15
rk92rk92
82
$endgroup$
add a comment |
$begingroup$
The questions are based on the below screenshots.
Can somebody explain how the functional Taylor expansion is related to a "standard" function Taylor expansion? In particular, I am concerned with this term
$$
C(F+ epsilon f) = C(F) + epsilon <nabla C(F), f>
$$
where $<cdot{}, cdot{}>$ is some suitable inner product.Why is it in general not possible to choose $f = - nabla C(F)$?
Source: Functional Gradient Descent for combining hypotheses by Mason et al. (1999)
functional-analysis taylor-expansion gradient-descent
edited Dec 5 '18 at 9:22
José Carlos Santos
157k22126227
asked Dec 5 '18 at 9:15
rk92rk92
82
$endgroup$
The questions are based on the below screenshots.
Can somebody explain how the functional Taylor expansion is related to a "standard" function Taylor expansion? In particular, I am concerned with this term
$$
C(F+ epsilon f) = C(F) + epsilon <nabla C(F), f>
$$
where $<cdot{}, cdot{}>$ is some suitable inner product.Why is it in general not possible to choose $f = - nabla C(F)$?
Source: Functional Gradient Descent for combining hypotheses by Mason et al. (1999)
functional-analysis taylor-expansion gradient-descent
functional-analysis taylor-expansion gradient-descent
edited Dec 5 '18 at 9:22
José Carlos Santos
157k22126227
asked Dec 5 '18 at 9:15
rk92rk92
82
edited Dec 5 '18 at 9:22
José Carlos Santos
157k22126227
asked Dec 5 '18 at 9:15
rk92rk92
82
edited Dec 5 '18 at 9:22
José Carlos Santos
157k22126227
edited Dec 5 '18 at 9:22
José Carlos Santos
157k22126227
edited Dec 5 '18 at 9:22
José Carlos Santos
157k22126227
157k22126227
asked Dec 5 '18 at 9:15
rk92rk92
82
asked Dec 5 '18 at 9:15
rk92rk92
82
asked Dec 5 '18 at 9:15
rk92rk92
82
82
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
It's analogous to a Taylor expansion provided you define a notion of continuity and functional derivative (like Gateaux or Frechet derivatives). Once you define such concepts given a functional with some properties you can derive a Taylor expansion (first order in the case you proposed) in the same way you would do for a normal real valued function.
Not sure about the question here, but when you have a functional you want to minimize you want to find its stationary points (as necessary condition), something like this leads to
$$
nabla C(F) = 0
$$
you can either solve this equation in closed form, if you can, or using a gradient flow (continuous version of gradient descent). If $F$ is your unknown function the gradient flow takes the form
$$
partial_tF = - nabla C(F)
$$
answered Dec 5 '18 at 14:51
user8469759user8469759
1,4011617
$endgroup$
$begingroup$
Thanks for your answer re 1. So you mean e.g. for Taylor Expansion of the form $f(a+h) = f(a) + f´(a)cdot{}h$ this would translate in my question from above (informally) to $f(cdot{}) = C(cdot{})$, $a=F$ and $h = epsilon f$.
$endgroup$
– rk92
Dec 5 '18 at 16:46
$begingroup$
Yes, but I think you could see a better resemblance with Taylor expansion if you consider a multivariate function $f(x), x in mathbb{R}^n$ + directional derivative. In such a case you would end up with: $$f(x + epsilon u) = f(x) + leftlangle nabla f, u rightrangle epsilon.$$ The expression is the same as your one, however what is "unclear" is the meaning of each symbol, to give those a meaning you need to use the concepts I mentioned in my answer.
$endgroup$
– user8469759
Dec 5 '18 at 16:51
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3026848%2ffunctional-gradient-descent-and-functional-taylor-expansion%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
It's analogous to a Taylor expansion provided you define a notion of continuity and functional derivative (like Gateaux or Frechet derivatives). Once you define such concepts given a functional with some properties you can derive a Taylor expansion (first order in the case you proposed) in the same way you would do for a normal real valued function.
Not sure about the question here, but when you have a functional you want to minimize you want to find its stationary points (as necessary condition), something like this leads to
$$
nabla C(F) = 0
$$
you can either solve this equation in closed form, if you can, or using a gradient flow (continuous version of gradient descent). If $F$ is your unknown function the gradient flow takes the form
$$
partial_tF = - nabla C(F)
$$
answered Dec 5 '18 at 14:51
user8469759user8469759
1,4011617
$endgroup$
$begingroup$
Thanks for your answer re 1. So you mean e.g. for Taylor Expansion of the form $f(a+h) = f(a) + f´(a)cdot{}h$ this would translate in my question from above (informally) to $f(cdot{}) = C(cdot{})$, $a=F$ and $h = epsilon f$.
$endgroup$
– rk92
Dec 5 '18 at 16:46
$begingroup$
Yes, but I think you could see a better resemblance with Taylor expansion if you consider a multivariate function $f(x), x in mathbb{R}^n$ + directional derivative. In such a case you would end up with: $$f(x + epsilon u) = f(x) + leftlangle nabla f, u rightrangle epsilon.$$ The expression is the same as your one, however what is "unclear" is the meaning of each symbol, to give those a meaning you need to use the concepts I mentioned in my answer.
$endgroup$
– user8469759
Dec 5 '18 at 16:51
add a comment |
$begingroup$
It's analogous to a Taylor expansion provided you define a notion of continuity and functional derivative (like Gateaux or Frechet derivatives). Once you define such concepts given a functional with some properties you can derive a Taylor expansion (first order in the case you proposed) in the same way you would do for a normal real valued function.
Not sure about the question here, but when you have a functional you want to minimize you want to find its stationary points (as necessary condition), something like this leads to
$$
nabla C(F) = 0
$$
you can either solve this equation in closed form, if you can, or using a gradient flow (continuous version of gradient descent). If $F$ is your unknown function the gradient flow takes the form
$$
partial_tF = - nabla C(F)
$$
answered Dec 5 '18 at 14:51
user8469759user8469759
1,4011617
$endgroup$
$begingroup$
Thanks for your answer re 1. So you mean e.g. for Taylor Expansion of the form $f(a+h) = f(a) + f´(a)cdot{}h$ this would translate in my question from above (informally) to $f(cdot{}) = C(cdot{})$, $a=F$ and $h = epsilon f$.
$endgroup$
– rk92
Dec 5 '18 at 16:46
$begingroup$
Yes, but I think you could see a better resemblance with Taylor expansion if you consider a multivariate function $f(x), x in mathbb{R}^n$ + directional derivative. In such a case you would end up with: $$f(x + epsilon u) = f(x) + leftlangle nabla f, u rightrangle epsilon.$$ The expression is the same as your one, however what is "unclear" is the meaning of each symbol, to give those a meaning you need to use the concepts I mentioned in my answer.
$endgroup$
– user8469759
Dec 5 '18 at 16:51
add a comment |
$begingroup$
It's analogous to a Taylor expansion provided you define a notion of continuity and functional derivative (like Gateaux or Frechet derivatives). Once you define such concepts given a functional with some properties you can derive a Taylor expansion (first order in the case you proposed) in the same way you would do for a normal real valued function.
Not sure about the question here, but when you have a functional you want to minimize you want to find its stationary points (as necessary condition), something like this leads to
$$
nabla C(F) = 0
$$
you can either solve this equation in closed form, if you can, or using a gradient flow (continuous version of gradient descent). If $F$ is your unknown function the gradient flow takes the form
$$
partial_tF = - nabla C(F)
$$
answered Dec 5 '18 at 14:51
user8469759user8469759
1,4011617
$endgroup$
It's analogous to a Taylor expansion provided you define a notion of continuity and functional derivative (like Gateaux or Frechet derivatives). Once you define such concepts given a functional with some properties you can derive a Taylor expansion (first order in the case you proposed) in the same way you would do for a normal real valued function.
Not sure about the question here, but when you have a functional you want to minimize you want to find its stationary points (as necessary condition), something like this leads to
$$
nabla C(F) = 0
$$
you can either solve this equation in closed form, if you can, or using a gradient flow (continuous version of gradient descent). If $F$ is your unknown function the gradient flow takes the form
$$
partial_tF = - nabla C(F)
$$
answered Dec 5 '18 at 14:51
user8469759user8469759
1,4011617
answered Dec 5 '18 at 14:51
user8469759user8469759
1,4011617
answered Dec 5 '18 at 14:51
user8469759user8469759
1,4011617
answered Dec 5 '18 at 14:51
user8469759user8469759
1,4011617
1,4011617
$begingroup$
Thanks for your answer re 1. So you mean e.g. for Taylor Expansion of the form $f(a+h) = f(a) + f´(a)cdot{}h$ this would translate in my question from above (informally) to $f(cdot{}) = C(cdot{})$, $a=F$ and $h = epsilon f$.
$endgroup$
– rk92
Dec 5 '18 at 16:46
$begingroup$
Yes, but I think you could see a better resemblance with Taylor expansion if you consider a multivariate function $f(x), x in mathbb{R}^n$ + directional derivative. In such a case you would end up with: $$f(x + epsilon u) = f(x) + leftlangle nabla f, u rightrangle epsilon.$$ The expression is the same as your one, however what is "unclear" is the meaning of each symbol, to give those a meaning you need to use the concepts I mentioned in my answer.
$endgroup$
– user8469759
Dec 5 '18 at 16:51
add a comment |
$begingroup$
Thanks for your answer re 1. So you mean e.g. for Taylor Expansion of the form $f(a+h) = f(a) + f´(a)cdot{}h$ this would translate in my question from above (informally) to $f(cdot{}) = C(cdot{})$, $a=F$ and $h = epsilon f$.
$endgroup$
– rk92
Dec 5 '18 at 16:46
$begingroup$
Yes, but I think you could see a better resemblance with Taylor expansion if you consider a multivariate function $f(x), x in mathbb{R}^n$ + directional derivative. In such a case you would end up with: $$f(x + epsilon u) = f(x) + leftlangle nabla f, u rightrangle epsilon.$$ The expression is the same as your one, however what is "unclear" is the meaning of each symbol, to give those a meaning you need to use the concepts I mentioned in my answer.
$endgroup$
– user8469759
Dec 5 '18 at 16:51
$begingroup$
Thanks for your answer re 1. So you mean e.g. for Taylor Expansion of the form $f(a+h) = f(a) + f´(a)cdot{}h$ this would translate in my question from above (informally) to $f(cdot{}) = C(cdot{})$, $a=F$ and $h = epsilon f$.
$endgroup$
– rk92
Dec 5 '18 at 16:46
$begingroup$
Thanks for your answer re 1. So you mean e.g. for Taylor Expansion of the form $f(a+h) = f(a) + f´(a)cdot{}h$ this would translate in my question from above (informally) to $f(cdot{}) = C(cdot{})$, $a=F$ and $h = epsilon f$.
$endgroup$
– rk92
Dec 5 '18 at 16:46
$begingroup$
Yes, but I think you could see a better resemblance with Taylor expansion if you consider a multivariate function $f(x), x in mathbb{R}^n$ + directional derivative. In such a case you would end up with: $$f(x + epsilon u) = f(x) + leftlangle nabla f, u rightrangle epsilon.$$ The expression is the same as your one, however what is "unclear" is the meaning of each symbol, to give those a meaning you need to use the concepts I mentioned in my answer.
$endgroup$
– user8469759
Dec 5 '18 at 16:51
$begingroup$
Yes, but I think you could see a better resemblance with Taylor expansion if you consider a multivariate function $f(x), x in mathbb{R}^n$ + directional derivative. In such a case you would end up with: $$f(x + epsilon u) = f(x) + leftlangle nabla f, u rightrangle epsilon.$$ The expression is the same as your one, however what is "unclear" is the meaning of each symbol, to give those a meaning you need to use the concepts I mentioned in my answer.
$endgroup$
– user8469759
Dec 5 '18 at 16:51
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3026848%2ffunctional-gradient-descent-and-functional-taylor-expansion%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
