Field is empty
'%1' is not a valid email address.
Field must not be empty.
It must be at least %1 characters long.
It must be less than %1 characters long.
It must be the same as the field '%1' value.
Invalid symbol entered. Allowed symbols:'%1'.
Number expected.
Positive number expected.
Integer expected.
Positive integer expected.
Values must be in the range [%1 .. %2].
Symbol %1 is already entered in allowed character set.
Field value must be less than %1.
First character must be a Latin letter.
Sun
Mon
Tue
Wed
Thu
Fri
Sat
January
February
March
April
May
June
July
August
September
October
November
December
century
B.C.
%1 century
Error occurred while importing data on line:%1. Invalid value: '%2'. Error: %3
"Could not determine a data field separator. You may use one of the following characters to separate data fields: tab, ';' or ','."
%2/%3/%1%4
%2/%3/%1%4 %6:%7
N
S
E
W
yes
no
Invalid file format. We support only the following formats: %1
Please leave your email or phone number.
minutes
minutes
minute
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
minutes
hour
hours
hours
hours
hours
hours
hours
hours
hours
hours
hours
days
day
days
days
days
days
days
days
days
days
days
days
month
months
months
months
months
months
months
months
months
months
months
year
years
years
years
years
years
years
years
years
years
years
ago
%1 minutes ago
%1 minutes ago
%1 minute ago
%1 minutes ago
%1 minutes ago
%1 minutes ago
%1 minutes ago
%1 minutes ago
%1 minutes ago
%1 minutes ago
%1 minutes ago
%1 minutes ago
%1 minutes ago
%1 hour ago
%1 hours ago
%1 hours ago
%1 hours ago
%1 hours ago
%1 hours ago
%1 hours ago
%1 hours ago
%1 hours ago
%1 hours ago
%1 hours ago
%1 days ago
%1 day ago
%1 days ago
%1 days ago
%1 days ago
%1 days ago
%1 days ago
%1 days ago
%1 days ago
%1 days ago
%1 days ago
%1 days ago
%1 month ago
%1 months ago
%1 months ago
%1 months ago
%1 months ago
%1 months ago
%1 months ago
%1 months ago
%1 months ago
%1 months ago
%1 months ago
%1 year ago
%1 years ago
%1 years ago
%1 years ago
%1 years ago
%1 years ago
%1 years ago
%1 years ago
%1 years ago
%1 years ago
%1 years ago
HTML code for site/blog
Appearance
Sample
Close
Sofia Vasilievna
ProfessionalComputers

Shannon Entropy

This online calculator computes Shannon entropy for a given event probability table and for a given message
Timur2013-06-04 09:40:17
In information theory, entropy is a measure of the uncertainty in a random variable. In this context, the term usually refers to the Shannon entropy, which quantifies the expected value of the information contained in a message.
The formula for entropy was introduced by Claude E. Shannon in his 1948 paper "A Mathematical Theory of Communication".

H(X)= - \sum_{i=1}^np(x_i)\log_b p(x_i)

Minus is used because for values less than 1 logarithm is negative, however, since

-\log a = \log \frac{1}{a},

formula can be expressed as

H(X)= \sum_{i=1}^np(x_i)\log_b \frac{1}{p(x_i)}

Expression
\log_b \frac{1}{p(x_i)}
is also called an uncertainty or surprisal, the lower the probability p(x_i), i.e. p(x_i) → 0, the higher the uncertainty or the surprise, i.e. u_i → ∞, for the outcome x_i.

Formula, in this case, expresses the mathematical expectation of uncertainty and that is why information entropy and information uncertainty can be used interchangeably.

There are two calculators below - one computes Shannon entropy for given probabilities of events, second computes Shannon entropy for given symbol frequencies for given message.

Shannon EntropyCreative Commons Attribution/Share-Alike License 3.0 (Unported)
Event probability table:
Event probability table
Save Cancel
Import data.
"One of the following characters is used to separate data fields: tab, semicolon (;) or comma(,)": 
OK Cancel
Add Import data. Clear table
 Entropy, bits:


Shannon EntropyCreative Commons Attribution/Share-Alike License 3.0 (Unported)
 Entropy, bits:


Not suitable?

View all calculators
(240 calculators in total. )
Request a calculator
 

Comments

Spam filter
Send comment