In 1856, he and his wife Mary Bronson LeDuc, with their two daughters moved downriver to Hastings, where they had also acquired two wheat farms and 160 acres with a small grist mill on the falls of the Vermillion River.

William and Mary chose three Downing designs for their rural home site; their residence, Carriage Barn and Ice House.

In 1958 he donated the home and outbuildings with 4 acres of land to the Minnesota Historical Society, with the agreement that he would continue using it for his antique business until he retired, which wasn't until 1986.

Citizen groups actively sought out ways to fulfill Carroll Simmons' vision of having the LeDuc house open to the public.

In 2002 the Minnesota Legislature apportioned 1.2 million dollars in bonding funds for MHS to preserve the house and bring it up to current building codes.

As part of the agreement, the Society transferred $604,000 of net assets provided by Carroll Simmons for endowment, repairs and maintenance of the LeDuc House.