The son learned the father's trade; at fifteen became a draughtsman and sign-painter; then worked for a lithographer; opened a studio and painted some ecclesiastical pictures.
In 1841 Rimmer made a tour of New England painting portraits; he lived in Randolph, Massachusetts, in 1845–1855 as a shoemaker; and, having studied with a respected physician, he practiced medicine from about 1848 to about 1860.
Through the intervention of a friend, Rimmer exhibited a plaster copy (later destroyed) of the Falling Gladiator in 1863 in the Salon des Refusés in Paris, France, where it impressed visitors with its unusually realistic anatomy.
[3] After teaching human anatomy and art in the Boston area, Rimmer became an influential teacher and director of the Cooper Union School of Design for Women in New York City.
In his final years, he taught at the school of the Museum of Fine Arts, Boston, where his lectures, as usual, were illustrated with much-admired blackboard sketches.
Rimmer's most famous work, though not usually associated with him, is Evening: Fall of Day, which was the prototype for the Swan Song Records logo that the English rock group Led Zeppelin used.