Born in Mitau, Latvia, the twelfth of thirteen children, Hirshhorn emigrated to the United States with his widowed mother at the age of six.

[3] A shrewd investor, he sold off his Wall Street investments two months before the collapse of 1929, realizing $4 million in cash.

[4][5] In the 1950s, he and geologist Franc Joubin were primarily responsible for the "Big Z" uranium discovery in northeastern Ontario and the subsequent founding of the city of Elliot Lake.

[6] From 1961 to 1976, Hirshhorn lived in a three-story Norman chateau in a 22-acre (89,000 m2) estate at the summit of Round Hill, a 550-foot (170 m) rise in north-central Greenwich, Connecticut, with a view of the Manhattan skyline.

Applying himself seriously to the study of art, he would question dealers, critics, and curators, and visit artists in their studios.

[8] Hirshhorn graced his Greenwich mansion with paintings by Joseph Glasco (and sculpture),[9] Willem de Kooning, Raphael Soyer, Jackson Pollock, Larry Rivers, and Thomas Eakins, and the grounds outside with sculptures by Auguste Rodin, Pablo Picasso, Henri Matisse, Alberto Giacometti, Alexander Calder, Richard Bernstein (artist), George Rickey, and Henry Moore.

[7] In 1966 Hirshhorn donated much of his collection, consisting of 6,000 paintings and sculptures from the 19th and 20th centuries (and constituting one of the world's largest private art treasures), to the United States government, along with a $2 million endowment.