KL divergence between two multivariate Bernoulli distribution
0
$begingroup$
The KL divergence between two Bernoulli distributions is:
$$
KL(p||q)_{Ber} = plog frac{p}{q} + (1-p)log frac{1-p}{1-q}
$$
According to my understanding, the KL divergence between two multivariate Bernoulli distributions $p$ and $q$ should be
$$
KL(p||q)_{Ber} = sum_{i=1}^{k} p_ilog frac{p_i}{q_i} + (1-p_i)log frac{1-p_i}{1-q_i}
$$
where k is the number of possible outcome.
Is this correct?
statistics probability-distributions machine-learning information-theory
asked Jan 14 '18 at 8:55
uchman21uchman21
1043
$endgroup$
|
show 2 more comments
0
$begingroup$
The KL divergence between two Bernoulli distributions is:
$$
KL(p||q)_{Ber} = plog frac{p}{q} + (1-p)log frac{1-p}{1-q}
$$
According to my understanding, the KL divergence between two multivariate Bernoulli distributions $p$ and $q$ should be
$$
KL(p||q)_{Ber} = sum_{i=1}^{k} p_ilog frac{p_i}{q_i} + (1-p_i)log frac{1-p_i}{1-q_i}
$$
where k is the number of possible outcome.
Is this correct?
statistics probability-distributions machine-learning information-theory
asked Jan 14 '18 at 8:55
uchman21uchman21
1043
$endgroup$
$begingroup$
Hi. Please consider learnin mathjax for typesetting. Images can be ok, but for typesetting it is usually discouraged.
$endgroup$
– mathreadler
Jan 14 '18 at 9:21
$begingroup$
Please do not use pictures for critical portions of your post. Pictures may not be legible, cannot be searched and are not view-able to some, such as those who use screen readers.
$endgroup$
– GNUSupporter 8964民主女神 地下教會
Jan 14 '18 at 12:16
1
$begingroup$
@mathreadler updated the equation format
$endgroup$
– uchman21
Jan 14 '18 at 12:41
$begingroup$
Great, looks much better.
$endgroup$
– mathreadler
Jan 14 '18 at 13:36
$begingroup$
"where $k$ is the number of possible outcome" A $k-$ multivariate Bernoulli doesn't have "$k$ possible outcomes" but $2^k$. Do we agree, or are we speaking of different things?
$endgroup$
– leonbloy
Jan 15 '18 at 21:30
|
show 2 more comments
0
0
0
$begingroup$
The KL divergence between two Bernoulli distributions is:
$$
KL(p||q)_{Ber} = plog frac{p}{q} + (1-p)log frac{1-p}{1-q}
$$
According to my understanding, the KL divergence between two multivariate Bernoulli distributions $p$ and $q$ should be
$$
KL(p||q)_{Ber} = sum_{i=1}^{k} p_ilog frac{p_i}{q_i} + (1-p_i)log frac{1-p_i}{1-q_i}
$$
where k is the number of possible outcome.
Is this correct?
statistics probability-distributions machine-learning information-theory
asked Jan 14 '18 at 8:55
uchman21uchman21
1043
$endgroup$
The KL divergence between two Bernoulli distributions is:
$$
KL(p||q)_{Ber} = plog frac{p}{q} + (1-p)log frac{1-p}{1-q}
$$
According to my understanding, the KL divergence between two multivariate Bernoulli distributions $p$ and $q$ should be
$$
KL(p||q)_{Ber} = sum_{i=1}^{k} p_ilog frac{p_i}{q_i} + (1-p_i)log frac{1-p_i}{1-q_i}
$$
where k is the number of possible outcome.
Is this correct?
statistics probability-distributions machine-learning information-theory
statistics probability-distributions machine-learning information-theory
asked Jan 14 '18 at 8:55
uchman21uchman21
1043
asked Jan 14 '18 at 8:55
uchman21uchman21
1043
edited Jan 15 '18 at 9:23
uchman21
asked Jan 14 '18 at 8:55
uchman21uchman21
1043
asked Jan 14 '18 at 8:55
uchman21uchman21
1043
asked Jan 14 '18 at 8:55
uchman21uchman21
1043
1043
$begingroup$
Hi. Please consider learnin mathjax for typesetting. Images can be ok, but for typesetting it is usually discouraged.
$endgroup$
– mathreadler
Jan 14 '18 at 9:21
$begingroup$
Please do not use pictures for critical portions of your post. Pictures may not be legible, cannot be searched and are not view-able to some, such as those who use screen readers.
$endgroup$
– GNUSupporter 8964民主女神 地下教會
Jan 14 '18 at 12:16
1
$begingroup$
@mathreadler updated the equation format
$endgroup$
– uchman21
Jan 14 '18 at 12:41
$begingroup$
Great, looks much better.
$endgroup$
– mathreadler
Jan 14 '18 at 13:36
$begingroup$
"where $k$ is the number of possible outcome" A $k-$ multivariate Bernoulli doesn't have "$k$ possible outcomes" but $2^k$. Do we agree, or are we speaking of different things?
$endgroup$
– leonbloy
Jan 15 '18 at 21:30
|
show 2 more comments
$begingroup$
Hi. Please consider learnin mathjax for typesetting. Images can be ok, but for typesetting it is usually discouraged.
$endgroup$
– mathreadler
Jan 14 '18 at 9:21
$begingroup$
Please do not use pictures for critical portions of your post. Pictures may not be legible, cannot be searched and are not view-able to some, such as those who use screen readers.
$endgroup$
– GNUSupporter 8964民主女神 地下教會
Jan 14 '18 at 12:16
1
$begingroup$
@mathreadler updated the equation format
$endgroup$
– uchman21
Jan 14 '18 at 12:41
$begingroup$
Great, looks much better.
$endgroup$
– mathreadler
Jan 14 '18 at 13:36
$begingroup$
"where $k$ is the number of possible outcome" A $k-$ multivariate Bernoulli doesn't have "$k$ possible outcomes" but $2^k$. Do we agree, or are we speaking of different things?
$endgroup$
– leonbloy
Jan 15 '18 at 21:30
$begingroup$
Hi. Please consider learnin mathjax for typesetting. Images can be ok, but for typesetting it is usually discouraged.
$endgroup$
– mathreadler
Jan 14 '18 at 9:21
$begingroup$
Hi. Please consider learnin mathjax for typesetting. Images can be ok, but for typesetting it is usually discouraged.
$endgroup$
– mathreadler
Jan 14 '18 at 9:21
$begingroup$
Please do not use pictures for critical portions of your post. Pictures may not be legible, cannot be searched and are not view-able to some, such as those who use screen readers.
$endgroup$
– GNUSupporter 8964民主女神 地下教會
Jan 14 '18 at 12:16
$begingroup$
Please do not use pictures for critical portions of your post. Pictures may not be legible, cannot be searched and are not view-able to some, such as those who use screen readers.
$endgroup$
– GNUSupporter 8964民主女神 地下教會
Jan 14 '18 at 12:16
1
1
$begingroup$
@mathreadler updated the equation format
$endgroup$
– uchman21
Jan 14 '18 at 12:41
$begingroup$
@mathreadler updated the equation format
$endgroup$
– uchman21
Jan 14 '18 at 12:41
$begingroup$
Great, looks much better.
$endgroup$
– mathreadler
Jan 14 '18 at 13:36
$begingroup$
Great, looks much better.
$endgroup$
– mathreadler
Jan 14 '18 at 13:36
$begingroup$
"where $k$ is the number of possible outcome" A $k-$ multivariate Bernoulli doesn't have "$k$ possible outcomes" but $2^k$. Do we agree, or are we speaking of different things?
$endgroup$
– leonbloy
Jan 15 '18 at 21:30
$begingroup$
"where $k$ is the number of possible outcome" A $k-$ multivariate Bernoulli doesn't have "$k$ possible outcomes" but $2^k$. Do we agree, or are we speaking of different things?
$endgroup$
– leonbloy
Jan 15 '18 at 21:30
|
show 2 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2604566%2fkl-divergence-between-two-multivariate-bernoulli-distribution%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2604566%2fkl-divergence-between-two-multivariate-bernoulli-distribution%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Hi. Please consider learnin mathjax for typesetting. Images can be ok, but for typesetting it is usually discouraged.
$endgroup$
– mathreadler
Jan 14 '18 at 9:21
$begingroup$
Please do not use pictures for critical portions of your post. Pictures may not be legible, cannot be searched and are not view-able to some, such as those who use screen readers.
$endgroup$
– GNUSupporter 8964民主女神 地下教會
Jan 14 '18 at 12:16
1
$begingroup$
@mathreadler updated the equation format
$endgroup$
– uchman21
Jan 14 '18 at 12:41
$begingroup$
Great, looks much better.
$endgroup$
– mathreadler
Jan 14 '18 at 13:36
$begingroup$
"where $k$ is the number of possible outcome" A $k-$ multivariate Bernoulli doesn't have "$k$ possible outcomes" but $2^k$. Do we agree, or are we speaking of different things?
$endgroup$
– leonbloy
Jan 15 '18 at 21:30