Built in 1766 and frequently extended and altered, it is one of Harrisburg's oldest buildings, and is nationally notable as the summer residence of Simon Cameron (1799–1889), an influential Republican Party politician during and after the American Civil War.

A long two-story ell extends the main block to the rear, giving the house an overall off-center T shape.

In 1853 it was adapted for use as the Pennsylvania Female College, which was forced into bankruptcy in 1861, due to turmoil caused by the American Civil War.

Cameron had recently resigned as United States Secretary of War, over a scandal involving fraudulent supply contracts.

He won election to the United States Senate in 1867, which helped consolidate his control over federal patronage money in Pennsylvania.