Claremont was home to the Temperance, Industrial, and Collegiate Institute, a school for African Americans founded by a formerly enslaved person.

English and Scotch emigrants had settled there, near the principal native American village called Quioughcohanock (probably located on what later became known as "Wharf Bluff") by 1632.

By the late 17th and early 18th centuries, Claremont was a busy port town and shipped many goods, but especially hogsheads of tobacco.

His principal heir, Lt. Col. William Allen (1768–1831), also lived at Claremont and served many terms in the House of Delegates before the War of 1812 and supposedly became the largest land owner and enslaver in the Commonwealth of Virginia of his day.

[8] He never married and required his great-nephew and heir, William Griffin Orgain to change his surname to Allen to inherit.

In the years since the area has remained a rural enclave, but some resort use has developed along the bluffs and beaches of the James River.

Many homes along the riverfront were severely damaged, and Claremont's neighboring beach, Sunken Meadow, was destroyed in 2003 by Hurricane Isabel.

Dr. John Jefferson Smallwood established the Temperance, Industrial, and Collegiate Institute in Claremont on October 12, 1892, with fewer than ten students.

The school's campus covered more than 65 acres along the James River and served boys and girls from Virginia as well as other states.

[6] The Richmond Planet covered the opening of Lincoln Memorial Hall on its campus in 1912, a significant accomplishment for the Institute.

[6] A prominent memorial to Smallwood may be found at the Abundant Life Church Cemetery in Spring Grove, Virginia.