Palmer was born in Detroit, where his mother was the daughter of the third Michigan Territorial Judge James Witherell, while his father was a New England merchant who had settled in the city following the War of 1812.

He traveled to Spain and South America and then entered the real estate business in Detroit in 1853 and then engaged in lumbering and agricultural pursuits with his future father-in-law, Charles Merrill, beginning in 1855.

Palmer was appointed United States Minister to Spain on March 12, 1889, by U.S. President Benjamin Harrison and served from June 17, 1889, to April 19, 1890.

Among his activities, Palmer was one of the major benefactors of the Michigan Soldiers and Sailors Monument erected at Campus Martius.

She bequeathed $3 million to found the Merrill-Palmer Institute in 1916, which is a national center for child and family development and is now affiliated with Wayne State University and located in the former house of Charles Lang Freer.

In 1885, the Palmers had had the prominent architecture firm of Mason & Rice design a rustic log cabin-style summer house on the land, which still remains in the park, although it is currently closed to visitors.