Entropy explained
In 1948 an engineer at Bell Labs, Claude Shannon, asked a question nobody had made precise: how much information does a message contain? His answer was a number, measured in bits, that depends not on what a message means but on how surprising it is. That number, entropy, sets the limits of every compression program, every network link and every storage device, and it explains why some texts shrink to a fraction of their size while others cannot be squeezed at all.
Information is surprise
Shannon's insight was to measure information by how unexpected a message is, not by what it means. If you already know the answer, hearing it tells you nothing; if it could have been one of a thousand things, it tells you a lot. A message that singles out one of two equally likely possibilities carries one bit. One of four takes two bits, one of eight three bits, and in general one of N equally likely possibilities takes log₂ N bits. A symbol that occurs with probability p therefore carries −log₂ p bits: the rarer it is, the more it tells you.
Shannon's formula
Entropy is the average information per symbol: H = −Σ p log₂ p, summed over every symbol with its probability p. For a fair coin H is 1 bit. For a coin that lands heads 90% of the time it is only 0.47 bits, because the outcome is usually no surprise. Entropy is largest when all symbols are equally likely, log₂ n bits for n symbols, and zero when one symbol is certain. Shannon chose the name on the advice of John von Neumann, because the same formula appears in physics, where entropy measures disorder.
The limit of compression
Shannon's source coding theorem says that no lossless code can use fewer bits per symbol, on average, than the entropy of the source, and that codes can come as close to it as you like. Huffman coding, invented by David Huffman in 1952 as a student, gives frequent symbols short codes and rare ones long codes and gets within one bit per symbol of the entropy; arithmetic coding gets closer still. Entropy is thus not just a description but a hard limit: a file of random bytes, with 8 bits of entropy per byte, cannot be compressed at all.
Context and redundancy
Letters in a text are not independent. After Q comes U, after a full stop a capital, and a word that starts with INFORMA almost certainly goes on with TION. Taking the previous symbols into account gives the conditional entropy, which is lower than the entropy of the symbols counted one by one. The difference is redundancy: the part of the message that could have been predicted. Compression programs such as ZIP, which uses the deflate method, find repeated strings and replace them with short references, and modern compressors predict each symbol from its context.
How much information is in English?
Counted letter by letter, with the frequencies of English, a letter carries about 4.1 bits, against 4.7 for letters chosen at random. In 1951 Shannon went further: he showed people the start of a text and asked them to guess the next letter, counting how many guesses they needed. From their answers he estimated that English carries only about 0.6 to 1.3 bits per letter. In other words, roughly three quarters of written English is redundancy, which is why a text with letters missing can usually still be read.
Randomness and passwords
Entropy also measures unpredictability, which is what security depends on. A key or password chosen uniformly from N possibilities has log₂ N bits of entropy, and an attacker must try about half of them on average. A random 12-letter password from the 26 lowercase letters has about 56 bits; a common word has far fewer, however long it is. Random number generators are tested by checking that their output has the full entropy of a random source, and operating systems collect entropy from timing noise to seed them.
Where entropy shows up
Entropy decides how many bits a phone call, a photo or a video stream needs, and how close a modem or Wi-Fi link can get to the capacity of its channel. Biologists use it to measure the information in DNA, where four bases can carry at most 2 bits each, and the diversity of an ecosystem. In machine learning, decision trees choose the question that reduces entropy most, and the cross-entropy loss trains most neural networks. Linguists compare languages by their entropy per character and per word.
Explore it
Open the entropy calculator to measure any text by characters, words or bytes. Compare a paragraph with random letters, bits, digits and DNA, see the surprise of every symbol, how much information the text holds against its stored size, and how the bits per symbol fall as more context is known, next to what ZIP actually achieves.
Frequently asked questions
Why is entropy measured in bits?
Because Shannon used logarithms to base 2, so that one yes-or-no choice between equal options counts as one unit. With the natural logarithm the unit is the nat, about 1.44 bits, and with base 10 the hartley, about 3.32 bits; physicists and some statisticians use those.
Can a text have more entropy than log₂ of its alphabet?
No. With n different symbols the entropy is at most log₂ n bits per symbol, reached only when all of them are equally common. Any imbalance, and any predictability from context, lowers it.
Is entropy in information theory the same as in physics?
The formulas are the same in shape: Boltzmann and Gibbs wrote physical entropy as a sum over the probabilities of a system's microscopic states. The meaning is related, as both measure how much you do not know, and physicists have since shown that erasing one bit of information must release a small amount of heat.
Why does compressing a compressed file not help?
A good compressor removes most of the redundancy, so its output is close to random, with almost 8 bits of entropy per byte. There is nothing left to remove, and compressing it again usually makes it slightly larger because of the compressor's own overhead.
Ready to try it?
Open PictorX in your browser — no install, no credit card required to start.
Open the entropy calculator