Questions tagged [maximum-entropy]
maximum entropy or maxent is a statistical principle derived from information theory. Distributions maximizing entropy (under some constraints) are thought to be "maximally uninformative" given the constraints. Maximum entropy can be used for multiple purposes, like choice of prior, choice of sampling model, or design of experiments.
173 questions
2
votes
1
answer
118
views
Intuition behind measure used in Steven Frank's "The common patterns of nature"
I have been working my way through The common patterns of nature by Steven Frank. I'm confused by the measure $m_y$ which he introduces in the section "The binomial distribution".
...[it] ...
- 163
5
votes
1
answer
142
views
Can conditioning eliminate VC dimension dependence in empirical process bounds?
I'm analyzing the function class:
$$
\mathcal{F} = \left\{ (x, z, y) \mapsto \mathbb{1}\{ y \leq z\alpha + x^\top\beta \} : \alpha \in \mathbb{R}, \beta \in \mathbb{R}^d \right\}.
$$
Let $\mathbb{G}_n(...
- 724
2
votes
1
answer
74
views
Which operations on distributions respect MaxEnt property?
Seems like MaxEnt property for log-normal distribution follows directly from MaxEnt property of normal. So for any Y, such that ...
- 133
2
votes
0
answers
39
views
Maximum entropy distributions with more general constraints
Gibbs showed that for a space $X$ (assume finite for simplicity) and functions $f_i:X\to R$, the maximum entropy distribution on $X$ s.t. constraints on the expectation of $f_i$ is the Boltzmann ...
- 3,077
1
vote
0
answers
57
views
Solve for maximum entropy conditional probability
I'm new to max-ent principle and functional derivatives. I have known joint data distribution $p_D(x,y)$ (where $y$ is regarded as the labels) and a latent variable model $(x,y,z)$ with the prior $p(z)...
- 307
1
vote
0
answers
55
views
Exponential families as families of limite distributions of Markov processes
An exponential family verifies a maximum entropy property: each density is the maximum entropy density given the expectation of its sufficient statistic.
On the other hand, from my understanding, the ...
- 151
1
vote
0
answers
47
views
Maximum Entropy distribution of a ticking clock
Say I have a clock that emits "ticks". An ideal clock looks like a dirac comb. It has:
perfect periodicity of ticks (there is a precise fixed time interval between any two consecutive ticks)...
- 277
0
votes
0
answers
133
views
Minimizing cross entropy over a restricted domain?
Suppose $f(x;q)$ is the true distribution. The support of the random variable $X$ is $\Omega$. Suppose, I am interested in a particular subset of $\Xi \subset \Omega$. I would like to minimize the ...
- 19
2
votes
1
answer
264
views
When and how was the Bernoulli distribution with real binomial proportion introduced?
I certainly should read Jakob Bernoulli's Ars Conjectandi again but let me share my concerns.
I'm just wondering when and how the Bernoulli distribution $Be(p)$ (and related distributions like the ...
- 39
5
votes
2
answers
238
views
Does every distribution family have a set of maximum entropy constraints?
I am reflecting on these examples of maximum entropy distributions. I am (pleasantly) surprised that various common distribution families have maximum entropy constraints.
It got me wondering if ...
- 10.1k
1
vote
1
answer
160
views
Is the principle of maximum entropy misleading?
If a distribution belongs to a certain class, then the distribution with the largest entropy in that class is typically referred to as the least-informative distribution. To me, this his highly ...
- 67
2
votes
1
answer
375
views
What is the reasoning behind max entropy constraints for the gamma distribution?
The max entropy method is a way of deriving a probability distribution given only the information you know about how the data is distributed and nothing more. For example the normal distribution can ...
- 214
8
votes
2
answers
687
views
Jaynes' Description of Maximum Entropy Distribution
I am reading E. T. Jaynes' probability theory book, and I am at chapter 11 where he introduces the maximum entropy principle. I understand that Jaynes separates the notion of probability from that of ...
- 187
1
vote
1
answer
187
views
How can we use shannon entropy to discriminate between two similar probability distribution function?
I studied two papers related to discriminating between two similar distributions using Shannon entropy. But both of them had different views. Can anyone explain what would be the basic flow of idea to ...
- 11
6
votes
1
answer
3k
views
Choosing "Target Entropy" for Soft-Actor-Critic (SAC) algorithm
I am quite familiar with Soft-Actor-Critic (SAC) and its many applications in continuous control RL environments. However, when implementing this algorithm in a practical setting, one thing that still ...
- 215