It is home to five historic buildings: Huntington Hall, the 1785 Rider House, the 1790 John Dodd Hat Shop, the Little Red Schoolhouse, and the Marian Anderson Studio.

[1] During the late 19th and 20th centuries, John Fanton, a Danbury industrialist and prominent citizen, and his second wife, Laura Scott, traveled extensively through Europe, Asia, and the Americas and collected artifacts from each country they visited.

In 1921, after their deaths, a museum was opened in the Fanton home on Deer Hill Avenue for the purpose of exhibiting these collections.

The purpose of the organization, for which the Rider house functioned as a meeting place, was to foster appreciation in art, science, history, music, and other public activities.

In 1947 a proposal was made that the Scott-Fanton Museum and the Danbury Historical Society and Arts Center merge.

During the 1950s and 1960s, the museum acquired the John Dodd Hat Shop, the Charles Ives House and erected Huntington Hall where many of its exhibits have been displayed.