Self-information

From Wikipedia, the free encyclopedia
  (Redirected from Information content)
Jump to navigation Jump to search

In information theory, self-information or surprisal is the surprise when a random variable is sampled. It is expressed in a unit of information, for example shannons (often called bits), nats, or hartleys, depending on the base of the logarithm used in its calculation. The expected self-information is information entropy and reflects the average surprise or uncertainty associated with sampling a random variable[1].

Definition[edit]

By definition, information is transferred from an originating entity possessing the information to a receiving entity only when the receiver had not known the information a priori. If the receiving entity had previously known the content of a message with certainty before receiving the message, the amount of information of the message received is zero.

For example, quoting a character (the Hippy Dippy Weatherman) of comedian George Carlin, “Weather forecast for tonight: dark. Continued dark overnight, with widely scattered light by morning.” Assuming one does not reside near the Earth's poles or polar circles, the amount of information conveyed in that forecast is zero because it is known, in advance of receiving the forecast, that darkness always comes with the night.

When the content of a message is known a priori with certainty, with probability of 1, there is no actual information conveyed in the message. Only when the advance knowledge of the content of the message by the receiver is less than 100% certain does the message actually convey information.

Accordingly, the amount of self-information contained in a message conveying content informing an occurrence of event, , depends only on the probability of that event.

for some function to be determined below. If , then . If , then .

Further, by definition, the measure of self-information is nonnegative and additive. If a message informing of event is the intersection of two independent events and , then the information of event occurring is that of the compound message of both independent events and occurring. The quantity of information of compound message would be expected to equal the sum of the amounts of information of the individual component messages and respectively:

.

Because of the independence of events and , the probability of event is

.

However, applying function results in

The class of function having the property such that

is the logarithm function of any base. The only operational difference between logarithms of different bases is that of different scaling constants.

Since the probabilities of events are always between 0 and 1 and the information associated with these events must be nonnegative, that requires that .

Taking into account these properties, the self-information associated with outcome with probability is defined as:

The smaller the probability of event , the larger the quantity of self-information associated with the message that the event indeed occurred. If the above logarithm is base 2, the unit of is bits. This is the most common practice. When using the natural logarithm of base , the unit will be the nat. For the base 10 logarithm, the unit of information is the hartley.

As a quick illustration, the information content associated with an outcome of 4 heads (or any specific outcome) in 4 consecutive tosses of a coin would be 4 bits (probability 1/16), and the information content associated with getting a result other than the one specified would be ~0.09 bits (probability 15/16). See below for detailed examples.

This measure has also been called surprisal, as it represents the "surprise" of seeing the outcome (a highly improbable outcome is very surprising). This term (as a log-probability measure) was coined[2] by Myron Tribus in his 1961 book[3] Thermostatics and Thermodynamics.

When the event is a random realization (of a variable) the self-information of the variable is defined as the expected value of the self-information of the realization.

Self-information is an example of a proper scoring rule.

Examples[edit]

  • On tossing a coin, the chance of 'tail' is 0.5. When it becomes known that 'tail' occurred, this amounts to
I('tail') = log2 (1/0.5) = log2 2 = 1 bit of information.
  • When throwing a fair dice, the probability of 'four' is 1/6. When it becomes known that 'four' has been thrown, the amount of self-information is
I('four') = log2 (1/(1/6)) = log2 (6) = 2.585 bits.
  • When, independently, two dice are thrown, the amount of information associated with {throw 1 = 'two' & throw 2 = 'four'} equals
I('throw 1 is two & throw 2 is four') = log2 (1/P(throw 1 = 'two' & throw 2 = 'four')) = log2 (1/(1/36)) = log2 (36) = 5.170 bits.
This outcome equals the sum of the individual amounts of self-information associated with {throw 1 = 'two'} and {throw 2 = 'four'}; namely 2.585 + 2.585 = 5.170 bits.
  • In the same two dice situation we can also consider the information asserted in the statement "The sum of the two dice is five"
I('The sum of throws 1 and 2 is five') = log2 (1/P('throw 1 and 2 sum to five')) = log2 (1/(4/36)) = 3.17 bits. The (4/36) is because there are four ways out of 36 possible to sum two dice to 5. This shows how more complex or ambiguous events can still carry information.

Relationship to entropy[edit]

The entropy is the expected value of the self-information of the values of a discrete random variable. Sometimes, the entropy itself is called the "self-information" of the random variable, possibly because the entropy satisfies , where is the mutual information of with itself.[4]

See also[edit]

References[edit]

  1. ^ Jones, D.S., Elementary information theory, Vol., Clarendon Press, Oxford pp 11-15 1979
  2. ^ R. B. Bernstein and R. D. Levine (1972) "Entropy and Chemical Change. I. Characterization of Product (and Reactant) Energy Distributions in Reactive Molecular Collisions: Information and Entropy Deficiency", The Journal of Chemical Physics 57, 434-449 link.
  3. ^ Myron Tribus (1961) Thermodynamics and Thermostatics: An Introduction to Energy, Information and States of Matter, with Engineering Applications (D. Van Nostrand, 24 West 40 Street, New York 18, New York, U.S.A) Tribus, Myron (1961), pp. 64-66 borrow.
  4. ^ Thomas M. Cover, Joy A. Thomas; Elements of Information Theory; p. 20; 1991.
  • C.E. Shannon, A Mathematical Theory of Communication, Bell Syst. Techn. J., Vol. 27, pp 379–423, (Part I), 1948.

External links[edit]