Relative Entropy

Let a discrete distribution have probability function , and let a second discrete distribution have probability function . Then the relative entropy of  with respect to , also called the Kullback-Leibler distance, is defined by

Although , so relative entropy is therefore not a true metric, it satisfies many important mathematical properties. For example, it is a convex function of , is always nonnegative, and equals zero only if .

Relative entropy is a very important concept in quantum information theory, as well as statistical mechanics (Qian 2000).

REFERENCES:

Cover, T. M. and Thomas, J. A. Elements of Information Theory. New York: Wiley, 1991.

Qian, H. "Relative Entropy: Free Energy Associated with Equilibrium Fluctuations and Nonequilibrium Deviations." 8 Jul 2000. http://arxiv.org/abs/math-ph/0007010.