Holder studied at the Friends' School in Providence, Rhode Island, and then entered the Harvard Medical School, where, while a student, he acted as demonstrator of anatomy for Oliver Wendell Holmes, and was present at the first administration of ether as an anesthetic.

[1]: 162–163 In 1859 he accepted the position of physician for Fort Jefferson,[2] Florida, at the request of Agassiz and Spencer F. Baird of the Smithsonian, and for seven years made an elaborate study of the Florida reef, being the first to establish the rapid growth of corals, an opposite view being held previous to that.

Holder resigned in 1869, to join Albert S. Bickmore in the establishment of the American Museum of Natural History of New York City, and became the curator of invertebrates.

He was the author of "The Florida Reef"; joint author with J. G. Wood of "Our Living World," an elaborate natural history; "The Museum of Natural History," written with Sir John Richardson; "The Atlantic Right Whale," and many scientific papers.

He was a patron of the Metropolitan Museum of Art; a founder and fellow of the American Ornithological Union; a member of the Society of Eastern Naturalists; member of the Society for Psychical Research; fellow of the New York Academy of Sciences, Geographical and Linnaen societies, and member of the Harvard Club.