Walker received an undergraduate degree in art history from Harvard University in 1930, where he studied with Paul J. Sachs.

He formed the Harvard Society for Contemporary Art, with Philip Johnson, Lincoln Kirstein, and Edward Warburg.

[2] He studied at Villa I Tatti in Florence with Bernard Berenson, and served as professor and assistant director of the American Academy in Rome from 1935 to 1939.

In 1956 he was named director of the National Gallery, succeeding David E. Finley Jr., and remained in the position until his retirement in 1969.

[2] During his tenure at the gallery, Walker cultivated donor relationships with collectors such as the Mellon family, Joseph Widener, Armand Hammer, and Chester Dale; his significant acquisitions included Rembrandt's Aristotle with the Bust of Homer, Fragonard's La Liseuse, El Greco's Laocoon, and the Ginevra de' Benci by Leonardo da Vinci.