entropy ← ##.shannon string                 ⍝ Shannon entropy of message ⍵.

From Gianfranco Alongi:  Shannon entropy is one of the most important metrics in
information theory.  Entropy measures the uncertainty associated with  a  random
variable, i.e. the expected value of the information in the message.

The concept was introduced by Claude E. Shannon  in  the  paper  "A Mathematical
Theory of Communication" (1948). Shannon entropy allows us to estimate the aver-
age minimum number of bits needed to encode a string of  symbols  based  on  the
alphabet size and the frequency of the symbols.

The entropy of the message is its amount of uncertainty;  it  increases when the
message is closer to random, and decreases when it is less random.

    Entropy of X ≡ H(X) := - sigma_i P(X_i) log_2(P(X_i))
    where
        P(X_i) = frequency of X_i in X

Ref: https://en.wikipedia.org/wiki/Information_theory#Entropy

Examples:

   (shannon⍤1 , ⊣) 1+~(⍳10) ∘.> ⍳10
0            2 2 2 2 2 2 2 2 2 2
0.4689955936 1 2 2 2 2 2 2 2 2 2
0.7219280949 1 1 2 2 2 2 2 2 2 2
0.8812908992 1 1 1 2 2 2 2 2 2 2
0.9709505945 1 1 1 1 2 2 2 2 2 2
1            1 1 1 1 1 2 2 2 2 2
0.9709505945 1 1 1 1 1 1 2 2 2 2
0.8812908992 1 1 1 1 1 1 1 2 2 2
0.7219280949 1 1 1 1 1 1 1 1 2 2
0.4689955936 1 1 1 1 1 1 1 1 1 2

See also: Data_compression

Back to: contents

Back to: Workspaces