Jouko Seppänen 2.4.2001Luonnonfilosofian seura - keskustelua informaatiosta
Kullervo Rainion valaisevasta puheenvuorosta jä puuttumaan algoritminen informaatioteoria (Kolmogorov), jonka mukaan bittijonon sisältämä informaation määrä on yhtä kuin lyhimmän mahdollisen algoritmin pituus, joka tarvitaan synnyttämään kyseinen bittijono.
Shannonin teorian mukaan säännöllinen (esimerkiksi pelkkiä ykkösiä sisältävä) ja yhtä pitkä 'satunnainen' bittijono sisältävät täsmälleen yhtä paljon informaatiota, kun taas Kolmogorovin mukaan säännöllinen on minimaalisen vähän ja satunnainen maksimaalisen paljon. Josko jotakuta kiinnostaa informaatioteorian historia yleisemmin, liitän oheen aihetta käsittelevän luvun katsauksestani 'History and Philosophy of System and Model Thinking, Information Theory and Cybernetics' (teoksessa: Systems - New Paradigms for the Human Sciences, Walter de Gruyter, Berlin, 1998). --
Information Theory
Information and Entropy
Classical Information Theory
Biological Information
Genetic Information
In discussing models above we have considered them as objects and means for obtaining knowledge about other objects. In doing so we have ignored the user of model, the subject or conscious agent. The distinction between object and subjectwas made already by Aristotle but the notion of subject still remains problematic. Using a model necessarily involves aconscious agent without which it makes no sense to speak about models. The same is true with the notion of information. Traditionally information is considered as existing objectively but, in fact, this is a mistake. Strictly speaking, information is a subjective concept although this aspect is usually ignored.
Information and knowledge involve, like models and theories, some entities like signs or symbols which stand not for themselves but for something else, namely for what they represent, i.e. what somebody aims them to mean. Therefore, they also require interpretation by a conscious agent. To be able to constructively define concepts like aim, intention, purpose etc. which are essential to organisms and concepts like sign, meaning, interpretation etc. which are essential to conscious beings and culture, it is necessary to define the notion information precisely. Once information has been defined as a subjective concept it becomes possible to consider more complex notions like learning, thinking, communication etc. which are based on it.
The word information, lat. in + forma, give form, was known to Roman philosopher Cicero (106-43) in the sense of conception or idea in human mind. Catholic church father St. Augustin (354-430) used the word in the sense of teaching and dissemination of biblical knowledge as did Thomas Aquinas (1225-1274). Later philosophers did not pay attention to the concept before than in the 20th century.
In the 20th century the notion of information became relevant in telecommunications engineering. A measure and mathematical theory of information became necessary for the analysis and design of reliable communication systems. Key ideas were pioneered in the early 1920's. In the thirties a relationship between entropy and information discovered by Hungarian-US physicist Leo Szilard (1898-1964) and the theory of information was formulated in the late 1940's by US mathematician and communication engineer Claude Shannon (1916-).
In the 1930's the notion of information was introduced also into mathematics by British statistician and geneticist Ronald A. Fischer (1890-1962), the founder of mathematical statistics. In the 1940's it was discovered as relevant to biology, specifically genetics where its significance was anticipated by German-US physicist and microbiologist Max Delbrück(1906-1981). The idea of genetic code was proposed and explored by Austrian physicist Erwin Schrödinger (1887-1961) in his book (1944) which signalled a new paradigm in the subsequent development of molecular biology which led to the discovery of the structure of DNA and RNA.
Toward the end of the century it is becoming clear that information is not a physical concept and can be defined precisely only by taking into account also the conscious subject for whome something is information, an observer who interprets a physical object or event as representing something else than what it in itself is, i.e. a sign or symbol standing for its signified in the mind of the conscious agent.
Information and Entropy
Harry Nyquist (1889-1976) - Communication and Information
Ralph V. Hartley (1928) - Logarithmic Information
Leo Szilard (1898-1964) - Information and Entropy
Léon Brillouin (1889-1969) - Information as Negentropy
In the beginning of the 20th century the notion of information was introduced in the context of telecommunication engineering and given a quantifying definition. In the 1930's analogies between information and entropy were noticed which led to the revival of thermodynamics and the dilemma of Maxwell's demon, a philosophical debate about the nature and relation of information to the physics, life and thermodynamics.
Harry Nyquist (1889-1976) - Communication and Information
US telegraph and telephone engineer Harry Nyquist (1889-1976) adopted the idea of information, for which he used the word 'intelligence', and applied it to the analysis of communication systems at the Bell Telephone Laboratories. He defined information as the rate per character at which 'intelligence' can be transmitted and gave it a measure as the logarithm of the number of possible distinct characters which could have been transmitted (1924).
1. Nyquist H. (1924): Certain factors affecting telegraph speed. Bell System Technical Journal 3, 324-346.
Ralph V. Hartley (1928) - Logarithmic Information
His colleague Ralph V. Hartley (1928) defined the measure of information in a message more generally as the logarithm of the number of messages which might might have been sent using a given alphabet of symbols. For instance, if a message consists of a sequence of n choices from k symbols, then the number of possible mesages of length k is kn and the amount of information contained in one message H = n log k, which is known as the logarithmic measure of information.
1. Hartley R.V.L. (1928): Transmission of Information. Bell System Technical Journal 7, 535-536.
Leo Szilard (1898-1964) - Information and Entropy
Hungarian-born US physicist Leo Szilard (1898-1964) noticed a connection between the concept of entropy in thermodynamics and information (1929). He defined the set of all possible messages made out of n symbols taken from an alphabet of k symbols as analoguos to the quantity of entropy which served as a reference for the measure of the amount of information conveyed by one message selected form the set. Szilard also conceived the idea of 'bit' as a unit of information equivalent to k ln 2 units of entropy. The term bit, short for binary digit, 0 or 1, was suggested about fifteen years after Szilard's work by US electrical engineer John Tukey.
1. Szilard L. (1929): On the decrease of entropy in a thermodynamic system by the intervention of
intelligent beings. Zeitschrift für Physik, 53:840-856.
Leon Brillouin (1889-1969) - Information as Negentropy
In the late fourties French-US physicist Léon Brillouin (1889-1969) noticed that the conceptual analogy between information and entropy was, in fact, mistaken because information was actually the opposite of entropy, i.e. H = -n log k. To emphasize this he introduced the term negentropy, short for negative entropy and investigated the question of life and its relation to thermodynamics and cybernetics (1949).
1. Brillouin L. (1949): Life, thermodynamics and cybernetics. American Scientist 37, 554-568.
Brillouin also analyzed and solved the dilemma of Maxwell's demon (1951). He showed that it was impossible for the demon to violate the entropy law, i.e. to extract energy from heat in a closed chamber by an 'intelligent demon', i.e. by using information to create potential differences among the molecules which move to opposite directions at different velocities. His argument was that in an enclosure at constant temperature the radiation is that of black body radiation, i.e. in balance, and the demon cannot see the molecules and hence, cannot operate the trap door.
2. Brillouin L. (1951): Maxwell's demon cannot operate: Information and Entropy. I. Journal of Applied Physics 22, 334-337.
Later it has been shown that even if the demon could do this the energy gained would be at most equal to the amount of energy needed to measure and use the information to operate the trap door or, due to the entropy law, less than what is gained. Therefore, the idea of a perpetuum mobile of the second kind, i.e. a machine that would be able to extract energy from even temperature, is impossible. The fallacy was shown to follow from the Platonic illusion of information as an immaterial (non-energetic) notion.
This result applies to all systems, natural and artificial, nonliving and living, including life and mind. By means of information, or more generally by means of making use of existing forms, order and structures, it possible to channel, direct and control the flow of energy in systems in such a way that the consumption and loss of free energy as dissipation becomes minimized. How this is possible will be discussed in the section on cybernetics. Moreover, free energy can even be accumulated and become transformed into new forms, structures and functions which can make still better use of energy as in self-organizing and developmental, learning and intelligent systems. These principles will be discussed in respective sections later.
Classical Information Theory
Claude E. Shannon (1916-) - Statistical Information Theory
Andrei N. Kolmogorov (1903-1987) - Algorithmic Information
Howard H. Pattee (1977) - Information and Measurement
Tom Stonier (1927-) - Information Physics
The early studies of information and communication revealed properties of information and its relationships with energy and entropy. In the light of physics the relation between the energy and information seems obvious but is still a problematic question. Physically information is modulated energy or matter but plays in the human context a different role than 'normal' physical processes. This role is characterized by the notion of measurement or conversion of energy or matter into a 'higher' form which is then used for purposes of communication, control, computing, reading, writing, copying etc. and on the level of physiology, neurology and psychology for perception, observation, learning, cognition, expression, interpretation etc. by conscious agents like animals and human beings.
On the other hand, physical, chemical and biological systems or their subsystems and functions can be seen as information processing systems and analyzed in terms of communication and control as is done in communication and control engineering, analysis of equilibrium dynamics and control of chemical reaction kinetics, especially enzyme chemistry and immunological reactions, and in various other biological contexts like genetic control, neurotransmission, pheromene communication etc. On the mental and social levels the role of information becomes even more significant but also more complicated as in systems and processes which involve the thought, language, learning, meaning and semiotics.
The study of all of thsese complex phenomena and systems can benefit from and preassume theories of information. I use the plural because there are several different theories of information. The classical information theory, also known as thestatistical information theory, was formulated by US mathematician Claude E. Shannon (1916-) in the late 1940's based on the notion of lograithmic information and notions of communication. In the 1960's Russian mathematician Andrei N. Kolmogorov (1903-1987) gave a computational definition to the measure of information in terms of the notion ofalgorithm which is known as the algorithmic information theory. Still other theories such as those of structural, semanticand dynamic information were formulated by philosophers, logicians, linguists, semioticians as well as by system scientists, cyberneticians and computer scientists.
In the 1970's the notion of information became subject of reconsideration in the light of philosophy of physics and physical measurement and was finally defined as a complex relative, subjective and systemic concept which necessarily involves a conscious agent and world model without which it is meaningless to speak about information.
Claude E. Shannon (1916-) - Statistical Information Theory
In the late 1930's US telecommunication engineer and mathematician Claude E. Shannon (1916-) discovered how the three basic logic operations, 'not', 'and', and 'or', could be done by means of three elementary electical switching circuits where by he laid foundation to the theory of switching logic, also called switching algebra (1938). This invention made possible the subsequent development of logic circuits and electronic digital computers.
1. Shannon C.E. (1938): A Symbolic analysis of relay and switching circuits. Transactions of the
American Institute of Electrical Engineers, 57:713-723.
Ten years later he developed a mathematical theory of information, also known as the classical or statistical of informationand communication (1948).
2. Shannon (1948): The Mathematical Theory of Communication. University of Illinois Press,
Urbana.
As basic concepts he adopted the notions of logarithmic measure of information and information entropy. By means of these he defined concepts of communication like sender, communication channel, receiver, signal, code, message, coding, decoding, noise, redundancy etc. The capacity of a channel to carry information he stated in the form of the channel lawwhich says that in any channel the amount of information can only diminish or at most conserve during transmisson which is due to external and internal interferences in the system.
Information thus was a measure for the capacity of a code to represent messages and the capacity of a channel totransmit them and discern signals from noise. Redundancy he defined as the additional information included in the message which may be lost during transmission without loss of message information and which can be used, in the case of loss of messagege information, to restore the message. This idea was later developed in coding theory and the theory ofself-correcting codes.
Andrei N. Kolmogorov (1903-1987) - Algorithmic Information
Russian mathematician Andrei N. Kolmogorov (1903-1987) defined the quantity and measure of information in terms of the concept of algorithm, lat. algoritm, algorizm, < pers. al-Khwarizmi, introduced by the Persian mathematician al-Khwarizmi (c.800-c.850). An algorithm is a prescribed sequence of well-defined rules for the solution of a problem in a finite number of logical and algebraic steps, i.e. a program which defines a computation. According to Kolmogorov thequantity of information in a given sequence of symbols is equal to the amount of information in terms of Shannon of the description of the simplest algorithm which can produce it. The amount of information producible by an algorithm thus is always smaller or at most equal to the amount contained in its own description. Although many algorithms can generate infinitely long sequences, and even chaotic, i.e. non-periodic infinite sequences, the amount of information contained in the description of the generating algorithm is finite. 1. Kolmogorov A. N. (1965): Tri podhodki k kvantifikatsii informatsii. Problemy peredachi informatsii, Moskva.
More recently the the theory of algorithmic information and computational complexity have been developed by US computer scientist Gregory Chaitin (1987). He has also applied it to theoretical biology and proposed that biological complexity measured in terms of algorithmic information manifests itself as either highly ordered or as highly random but not as anything between these extremes.
2. Chaitin G. (1987): Information, Randomness and Incompleteness. World Scientific, Singapore.
The amount of information generated by an algorithm is never greater than the amount contained in its own description. The upper limit, when it becomes equal to its description, is achieved by a self-replicating algorithm, i.e. an algorithm which reproduces its own description and the logic of which underlies also the basic chemical mechanism of life on Earth. The theory of self-reproducing automata was pioneered by Hungarian-born US physicist, mathematician and computer scientist John von Neumann (1903-1957) in the 1950's (1966).
1. Neumann J. von (1966): The Theory of Self-reproducing Automata. Ed. by Burks A.W., University of
Illinois Press, Urbana.
Howard H. Pattee (1977) - Information and Measurement
In the methodology of exact science the role of observation and measurement are central. These are operations by which physical effects are converted into information, i.e. obserwed values, measurement data, quantities etc. Mathematical information theory ignores the physical aspects and ultimately physical nature of information. This situation led to a crisis in physics first noticed in quantum mechanics by Danish physicist Niels Bohr (1885-1962) in the 1920's.
US phycisist and system scientist Howard H. Pattee (1977) generalized the dilemma of quantum mechanics and demonstrated that if a measurement result is to be interpreted as information the interpretation must take place conceptually on a level higher than the event of measurement. In addition, it is necessary to consider the fact that the observing or measuring event always and inevitably has some causal connection with and ultimately some influence on the object of measurement. Moreover, this holds true equally for physical and technical measurements as for human perception.
1. Pattee H.H. (1977): Dynamic and Linguistic Models of Complex Systems. In International Journal of General Systems 3: 259-266.
To measure quantum phenomena one cannot avoid using ultimately other quantum effects as the means of measurement. Such a situation leads to unpredictable results. The situation can be compared with probing qualities of a billiard ball by using another billiard ball, i.e. the same level phenomena as the object of study and the means of measurement. As a consequence the difference between physics and information becomes blurred and predictability of the results disappears. The situation becomes even more critical if more than two objects interact. The collision of three billiard balls is as inpredictable as the there-body system in celestial mechanics and three frequency-system in oscillation dynamics.
Tom Stonier (1927-) - Information Physics
Also British physicist and information scientist Tom Stonier (1927-) posed the question 'What is Information' in a philosophical setting and developed a theory of what he calls information physics (1990). He examines the proposition that information has physical reality, like matter and energy, as a constituent of the universe rather than being a product of the human mind. He argues that information has existence independent of our ability to understand and use it. He considers the degree of organization as a measure of its information and related it to order and structure found in physics, chemistry, biology and higher systems including culture and technology. Moreover, he explores the possibility that information, like light, may ultimately exist in particle form, for which he proposes to be called infons in analogy to photons, electrons and other elementary particles. Infons do not show up in traditional physical experiments since they do not possess mass or energy but, according to Stonier, they manifest their effect as changes in organization.
1. Stonier T. (1990): Information and the Internal Structure of the Universe. An Exploration into
Information Physics. Springer-Verlag, Berlin.
None of the above theories of information say nothing about the possible meanings of the signals, signs or messages. Theories of meaning and other aspects of information have been formulated in logic, linguistics, semiotics, cybernetics and the theory of formal languages and algebraic semantics. But theories of meaning are still problematic since meaning depends not only on the form and content of the message but also on the contexts where it is expressed and interpreted as well as on the world and language models of the formulator and interpreter.
As a scientific notion information is considered as a fundamental category on par with matter and energy. It has proven useful for characterizing a wide range of phenomena of the non-living and living nature, especially on the level of mind, society and culture including science and technology, which would otherwize be difficult or impossible to cope with. Information is closely related with notions like probability, combination, selection, order, structure, complexity, organization, control, communication, learning, consciousness etc. It is also a prerequisite for the definition and precise discussion of these higher level notions. The definition of information is, however, still problematic and controversial since its relation to the nonliving physical nature and to the living, especially the human mind, are unadequately defined. This situation causes difficulties in using, defining and interpreting the notions of information in applied contexts such physical, chemical or as biological information in natural sciences and questions of concerning information and meaning in human and social sciences and cultural studies.
Biological Information
Alfred Kossel (1853-1927) - Nucleic Acids
Max Delbrück (1906-1981) - Exchange of Genetic Material
Phoebus Levene (1869-1940) - RNA, DNA and Nucleotides
Erwin Chargaff (1905-) - Base-pairing Rules
During the first decennia of the 19th century advances in biochemistry and molecular biology made possible the study of heredity and genetics by the methods of chemistry and physical chemistry. In the 1930's the role of nucleic acids in heredity was established with the discovery of RNA and DNA. In the fourties the base-pairing relations in DNA and in the fifties its stereo structure as a double helix were discovered. Meanwhile physicists, too, became involved in the study and speculation with the physico-chemical basis of life on the molecular level and introduced notions from physics, thermodynamics and information theory into biology.
In the 1960's new fields like bioenergetics and bioinformatics were formed which amply borrowed from mathematics, physics, chemistry and computer science, especially the theory of programming languages, data structures and algorithms as well as from linguistics. In the 1970's and eighties computer models became widely applied to biology. On the other hand biological notions and principles were borrowed into computer science and new interdisciplinary fields arose suchs as genetic algorithms, neurocomputing, artificial life etc. Yet, the notion of biological information remains problematic until today.
Alfred Kossel (1853-1927) - Nucleic Acids
During the first decennia of the 20th century the classical theory of heredity established by Austrian botanist Gregor Mendel (1822-1884) in 1856 had been revolutionized by the discovery of chromosomes and genes in the cell nucleus. Chemical analysis had revealed amino acids as the building blocks of proteins, and nucleic acids as the components of chromosomes and genes which were identified as the carriers of genetic material. In the 1920's German biochemist Alfred Kossel (1853-1927) analyzed nucleic acids chemically and gave them the names adenine (A), cytosine (C), guanine (G) and thymine (T).
Max Delbruck (1906-1981) - Exchange of Genetic Material
In the 1930's also physicists became involved in genetics and molecular biology. German-US microbiologist and biopysicistMax Delbrück (1906-1981) was active in both physics and biology became convinced that genetic problems were solvable by chemistry and physics. He insisted, however, that new concepts would be needed to cope with the complexity of biochemistry. He decided to study viruses and bacteria as the simpliest forms of life and discovered that they were able to exchange and recombine genetic material. Viruses called phagues, gr. phag, eat, could penetrate bacteria, inocculate them and overtake the control of their genetic machinery. These exchanges Delbrück saw as a form of communication of 'messages' and made him to think that molecular biology must have an 'informational basis', an idea that was to bear fruit soon.
Phoebus Levene (1869-1940) - RNA, DNA and Nucleotides
In the 1930's Russian-US biochemist Phoebus Levene (1869-1940) showed that nucleic acids were of two kinds and named them as ribonucleic acid (RNA) and deoxiribonucleic acid (DNA), and suggested that they were the basis of heredity on the molecular level. Moreover, Levene showed that the nucleic acids consisted structurally of units which had three chemical components, a phosphoric acid, a sugar and one purine or pyrimidine molecule, which together he called anucleotide.
Erwin Chargaff (1905-) - Base-pairing Rules
In the 1940's Czech-US biochemist Erwin Chargaff (1905-) discovered the base-pairing rules in DNA which concern the relative amounts of the four bases of nucleic acids, i.e. A=T and C=G, which were to be of great value as a cue in the analysis and explication of the chemical structure of RNA and DNA.
Genetic Information
Erwin Schrödinger (1887-1961) - Genetic Information
Cyril Hinselwood (1897-1967) - Nucleic Acids as Genetic Code
Francis Crick (1916-) - Double Helix Structure of DNA
George Gamow (1904-1968) - Triplet Structure of the Code
Robert Holley (1922-) - Decipherment of the Triplet Code
In the 1940's the notion of information and associated concepts were introduced into biology and given interpretation in terms of genetics and molecular biology. The double helix structure and principles of coding of amino acids in protein synthesis by nucleotide sequences were uncovered and allowed molecular biology to be discussed in terms of the theory of information, communication and computation.
Erwin Schrödinger (1887-1961) - Genetic Information
In 1944 Austrian physicist Erwin Schrödinger (1887-1961), the founder of wave mechanics, was inspired by Delbrück's idea and wrote the celebrated little book (1944) which signalled the beginning of a new age and paradigm shift in biology, especially genetics. He introduced the concept of genetic information and related ideas borrowed from physics, thermodynamics and the theory of communication including 'feeding on negative entropy' as a physical explanation of the ability of living beings to maintain their vital functions and to grow and develop, chromosome as a 'coded message' and DNA as an 'aperiodic crystal'. These metaphors turned out to be helpful in the subsequent study and discussion of the complex mechanisms and principles of molecular genetics which were soon to be uncovered.
1. Schrödinger E. (1944): What Is Life? The Physical Aspect of the Living Cell. Cambridge University Press, Cambridge.
Cyril Hinselwood (1897-1967) - Nucleic Acid as Genetic Information
In 1936 German organic chemist Hermann Staudinger (1881-1965) anticipated that genes are macromolecules with a definite structure which determines their functions in the cell. In 1950 British physical chemist Cyril Hinselwood (1897-1967) suggested that in the synthesis of proteins it is the nucleic acids which determine the order in which amino acids are linked together to form proteins.
Francis Crick (1916-) - Double Helix Structure of DNA
In 1953 British biochemist Francis Crick (1916-) and US molecular biologist James Watson (1928-) assisted and rivalled by British physicist Maurice Wilkins (1916-) and his colleague biochemist Rosalind Franklin (1920-1958) analyzed and solved the stereochemical structure of DNA as a double helix consisting of two nucleotide strands linked by bonds between the complementary bases according to Chargaff's pairing rules.
George Gamow (1904-1968) - Triplet Structure of the Code
In 1954 Russian-US physicist George Gamow (1904-1968) proposed that the four nucleic acids A, C, G and T constituted the genetic code or the code of life and that since they were only four whereas amino acids were 20 in number, a sequence of least three nucleotides as a block, which is called a codon, were needed to code for one amino acid.
Robert Holley (1922-) - Decipherment of the Triplet Code
The triplet nucleotide code was analyzed in detail in the 1960's by US biochemists Robert Holley (1922-) and Marshal Nirenberg (1927-) who deciphered 50 triplets, compiled a dictionary of the RNA code and translation machiner. Moreover, they found that certain amino acids could be specified by more than one triplet and some triplets did not specify an amino acid at all. These 'nonsense' triplets signified the beginning or end of a strand. It became also clear that the control of the order is a one-way process from the DNA code to proteins, which is known as the central dogma in the theory of evolution, while enzymes which are proteins act as catalysts in the processes of translation and transcription. Thus the interplay between nucleic acids and proteins is like that between hen and egg one determining the other.
|