It was built as a token of love from husband to wife, the depth of James and Sallie May's relationship being represented in the 4,000 piece Tiffany stained-glass window and a domed ceiling bearing the likeness of Mrs. Dooley [4] Despite the lavish expenditure, it was occupied only for a few years following completion in 1912.

Like Monticello, Thomas Jefferson's house 27 miles away, it had a dumbwaiter to bring food up from the basement kitchen to the butler's pantry on the first floor and placed on a radiator with flat shelving.

During the Country Club era, they built a small stone building on the property for guests to pay their golf fees and it was rumored to house the region's best moonshine distillery and to be a favored supplier for government officials during Prohibition.

[7] The United States Navy considered purchasing and renovating the property in 1942, which they calculated would cost $200,000, for the purpose of establishing a secret facility to interrogate prisoners of war.

The military rejected it in favor of a Civilian Conservation Corps camp in Fort Hunt, Virginia, because it seemed unlikely that Congress would approve the purchase of such a palatial structure for the purpose.

[8] The mansion stood empty through the Great Depression and World War II until A.T. Dulaney purchased it with a group of Charlottesville business men and formed Skyline Swannanoa, Inc.