On that evening Confederate General Robert E. Lee used the house as his headquarters, after being initially occupied that day by Federal troops of the Union Sixth Corps.

Idlewild was built in 1859 by William Yates Downman, a blue blooded Virginian whose family seat was Belle Isle in Lancaster County.

Their oldest daughter Anne Hayes (Nannie) Downman would grow up and marry R. Innes Taylor of nearby Fall Hill.

Rawleigh William Downman was the first son to be born at Idlewild but would pass away at the early age of twenty one while visiting relatives in Baltimore.

James Hayes Downman would become a successful New York businessman who came back to Fredericksburg where he died while visiting sister Nannie at her home at Fall Hill.

Mary Ann had the first Tiffany stained glass window in St. George's Church in Fredericksburg installed in memory of these two sons.

In later years, Mary Ann, Nannie and Sophia moved back to their "city home" on Caroline Street in Fredericksburg.

Mention is made of some of the incidents in the "practice" letters of daughter Sophia who jotted down her correspondence in her father's old notebooks - one volume of which recently found its way back to Fredericksburg.

The land directly surrounding the mansion remains intact with older milking sheds, corn cribs and other non-contributing structures in place.