Established in 1850, the 13.47-acre (5.45 ha) cemetery contains the graves of some of Knoxville's most influential citizens, ranging from politicians and soldiers, to artists and activists.

The cemetery is also noted for the Victorian era marble sculpture and elaborate carvings adorning many of the grave markers and headstones.

Unlike its crowded predecessor, the First Presbyterian Church Cemetery, Old Gray features spacious graves, grand monuments, and extensive vegetation, and its layout bears more resemblance to a public park.

[1] In February 1850, a board of trustees, led by East Tennessee University president William B. Reese, was appointed to buy land and sell lots for a new cemetery.

At the suggestion of Reese's wife, Henrietta, the cemetery was named after English poet Thomas Gray, author of Elegy Written in a Country Churchyard.

[4] The first burial had, however, occurred on July 15 of the previous year, after a local man named William Martin died of wounds from a cannon explosion during the city's Fourth-of-July celebration.

Martin's grave was not marked, but a small marble memorial in the northwest section of the cemetery recalls the incident.

Other notable markers include Woodmen of the World monuments—carved to depict undressed tree logs—where were given to the order's members as part of a life insurance policy.

[7] The family plots of two bitter Civil War rivals, pro-Unionist William "Parson" Brownlow and pro-secessionist John Hervey Crozier, are separated only by a roadway.