"A Logical Calculus of the Ideas Immanent to Nervous Activity" is a 1943 article written by Warren McCulloch and Walter Pitts.
[1] The paper, published in the journal The Bulletin of Mathematical Biophysics, proposed a mathematical model of the nervous system as a network of simple logical elements, later known as artificial neurons, or McCulloch-Pitts neurons.
These neurons receive inputs, perform a weighted sum, and fire an output signal based on a threshold function.
By connecting these units in various configurations, McCulloch and Pitts demonstrated that their model could perform all logical functions.
is the Heaviside step function (outputting 1 if the input is greater than or equal to 0, and 0 otherwise).
Language II covers substantial parts of classical mathematics, including real analysis and portions of set theory.
The definition above is spatial summation (which they pictured as having multiple synapses placed close together, so that the effect of their firing sums up).
may involve reference to past events of an indefinite degree of remoteness".
[4] As a remark, they noted that a neural network, if furnished with a tape, scanners, and write-heads, is equivalent to a Turing machine, and conversely, every Turing machine is equivalent to some such neural network.
Thus, these neural networks are equivalent to Turing computability, Church's lambda-definability, and Kleene's primitive recursiveness.
[5][6] In the symbolic logic side, it built on the previous work by Carnap, Whitehead, and Russell.
This was contributed by Walter Pitts, who had a strong proficiency with symbolic logic.
Pitts provided mathematical and logical rigor to McCulloch’s vague ideas on psychons (atoms of psychological events) and circular causality.
[7] In the neuroscience side, it built on previous work by the mathematical biology research group centered around Nicolas Rashevsky, of which McCulloch was a member.
The paper was published in the Bulletin of Mathematical Biophysics, which was founded by Rashevsky in 1939.
During the late 1930s, Rashevsky's research group was producing papers that had difficulty publishing in other journals at the time, so Rashevsky decided to found a new journal exclusively devoted to mathematical biophysics.
[8] Also in the Rashevsky's group was Alston Scott Householder, who in 1941 published an abstract model of the steady-state activity of biological neural networks.
The model, in modern language, is an artificial neural network with ReLU activation function.
[9] In a series of papers, Householder calculated the stable states of very simple networks: a chain, a circle, and a bouquet.
Walter Pitts' first two papers formulated a mathematical theory of learning and conditioning.
[10] In 1938, at age 15, Pitts ran away from home in Detroit and arrived in the University of Chicago.
He wrote several early papers on neuronal network modelling and regularly attended Rashevsky's seminars in theoretical biology.
The seminar attendants included Gerhard von Bonin and Householder.
[11] McCulloch had been interested in circular causality from studies with causalgia after amputation, epileptic activity of surgically isolated brain, and Lorente de Nò's research showing recurrent neural networks are needed to explain vestibular nystagmus.
[4][10] Both authors' affiliation in the article was given as "University of Illinois, College of Medicine, Department of Psychiatry at the Illinois Neuropsychiatric Institute, University of Chicago, Chicago, U.S.A." It was a foundational result in automata theory.
[14] McCulloch was the chair to the ten Macy conferences (1946--1953) on "Circular Causal and Feedback Mechanisms in Biological and Social Systems".
[15] In the 1943 paper, they described how memories can be formed by a neural network with loops in it, or alterable synapses, which are operating over time, and implements logical universals -- "there exists" and "for all".
This was generalized for spatial objects, such as geometric figures, in their 1947 paper How we know universals.
[16] Norbert Wiener found this a significant evidence for a general method for how animals recognizing objects, by scanning a scene from multiple transformations and finding a canonical representation.
[17] McCulloch worked with Manuel Blum in studying how a neural network can be "logically stable", that is, can implement a boolean function even if the activation thresholds of individual neurons are varied.