In addition to the former station building, the museum has a collection of heritage railcars in the neighboring rail yard it shares with Metro-North.

By 1993, that had dwindled to a few commuter trains, and the Connecticut Department of Transportation, which by then owned the neglected building, closed it in favor of a newer station on the other side of the block.

The museum itself is located on a 1.3-acre (0.53 ha) lot at the southeast corner of White Street and Patriot Drive, just across from Meeker's Hardware, which is also on the Register.

A grade crossing on White marks the eastern terminus of the Beacon Line kept in reserve by Metro-North for possible future use.

[1] The station building is a one-story L-shaped structure of buff and brown brick with sandstone trim, 99 by 123 feet (30 by 37 m), both wings topped with gabled roofs covered in asphalt.

Hipped-roof dormer windows pierce the north and west elevations, and similar canopies run along the tracks on either side, continuing the overhanging bracketed eaves that shelter the platform on the building itself.

[1] Inside, the museum's exhibits and displays occupy the 74-by-40-foot (23 by 12 m) southern half of the building, its former waiting room.

By the 1880s, the three railroads that served the city — the Danbury and Norwalk, Housatonic and New York and New England — had built small, separate stations for their lines in the vicinity of the current building.

Later in that decade, economic difficulties led to them all becoming part of the New York, New Haven and Hartford, leading it to be known locally as the Consolidated Road.

A. Malkin's design combined a basic Richardsonian Romanesque structure with some Colonial Revival details, like the small panes in the windows.

Other passengers were commuters going to jobs in Bridgeport or New York City, summer travelers headed for country retreats and hotels in the area, or visitors to the Danbury Fair, held every October.

[1] After World War II, train ridership began to decline with the rise of passenger air travel and the Interstate Highway System.

The city's mayor, Gene Eriquez, who had seen downtown wither as retail business and customers went to the mall built on the former fairgrounds, did not want to see another old building lost to urban blight.