The dissertation of Carleman under Erik Albert Holmgren, as well as his work in the early 1920s, was devoted to singular integral equations.

[5] At about the same time, he established the Carleman formulae in complex analysis, which reconstruct an analytic function in a domain from its values on a subset of the boundary.

[7] Later, he worked in the theory of partial differential equations, where he introduced the Carleman estimates,[8] and found a way to study the spectral asymptotics of Schrödinger operators.

[9] In 1932, following the work of Henri Poincaré, Erik Ivar Fredholm, and Bernard Koopman, he devised the Carleman embedding (also called Carleman linearization), a way to embed a finite-dimensional system of nonlinear differential equations du⁄dt = P(u) for u: Rk → R, where the components of P are polynomials in u, into an infinite-dimensional system of linear differential equations.

[13][14] Returning to mathematical physics in the 1930s, Carleman gave the first proof of global existence for Boltzmann's equation in the kinetic theory of gases (his result applies to the space-homogeneous case).

Carleman supervised the Ph.D. theses of Ulf Hellsten, Karl Persson (Dagerholm), Åke Pleijel and (jointly with Fritz Carlson) of Hans Rådström.

His interaction with William Feller before the former departure to the United States was not particularly pleasant, at some point being reported due to his opinion that "Jews and foreigners should be executed".

[21] Carlson remembers Carleman as: "secluded and taciturn, who looked at life and people with a bitter humour.