Noted for its castle-like appearance and eccentric, unplanned design, the building was home to a one-doctor hospital operated by retired U.S. Army physician Fred Stone Sr. (1887–1976) in the 1940s, 1950s, and 1960s.

[3] In 2006, the building was added to the National Register of Historic Places for its association with the region's medical services history, namely the transition from rural country doctors to modern hospitals.

[2] The grandson of a noted Claiborne County doctor, Stone spent his teen years drifting around the country before returning to East Tennessee to obtain his medical degree in 1916.

After travelling around the world, he returned to East Tennessee, where in the early 1940s he worked as an examiner for new Manhattan Project employees at Oak Ridge.

[3] In 1943, Stone purchased what was then a simple two-story building in Oliver Springs for use as a hospital, and spent the next three decades expanding it, adding multiple stories, hidden corridors, marble terraces, and a six-story central observation tower.

At the age of 19, Fred moved to New Mexico to homestead a small plot of land, and within a few years had joined several relatives in Oregon, where he worked in hops vineyards.

[3] Using money he had saved up, Stone returned to East Tennessee and enrolled in Lincoln Memorial University Medical School in Knoxville, from which he graduated in 1916.

After World War II, Stone devoted himself primarily to his Oliver Springs hospital, which he purchased in 1943, seeing patients at all hours of the day and expanding the building room-by-room in his spare time.

He hired a brick-mason named Joe Chittum and mobilized idle nurses and relatives to mix mortar and aide in bricklaying.

The building's central six-story tower, completed in 1949, briefly served as a Civil Defense lookout for the nearby atomic energy installations at Oak Ridge.

The building's subsequent owners removed the Maple tree and added fiberglass covers to parts of the roof to prevent water leakage.

Part of the rear section's northeast wall is recessed, as Stone attempted to build around a Maple tree growing on the lot.