Perry was the son of Henry Pierrepont Perry, a Wall Street stockbroker, and Edith Lounsbery, who was the daughter of Richard P. Lounsbery and Edith Hunter Haggin who in turn was the daughter of one of America's most prominent horsemen, James Ben Ali Haggin.
[2][3][4] Although born in New York City, Perry spent a good deal of his childhood at Rancho Del Paso, the huge Haggin family ranch in Sacramento County, California.
The family owned a summer estate in Newport, Rhode Island and young Perry studied there at St. George's School before going on to Yale University.
[5] In 1960, through his Gamely Corporation, Perry entered into an annual foal sharing partnership with Arthur Hancock of Claiborne Farm.
[7] Perry and his wife Nicole made their home at Waterford Farm, a 144-acre equestrian estate in Buchanan County, Virginia about four miles from the village of Middleburg.