Shortly afterwards Colt added 20 six/eight-family houses (10 of which survive) on Huyshope and Van Block Avenues for skilled workers.

Colt's 1855 East Armory was almost totally destroyed by a disastrous fire in 1864; only two small outbuildings remain of this original construction (the Forge and the Foundry).

The building is capped with a distinctive onion-shaped, sheet metal dome, painted deep blue with gold stars, and resembling that of the 1855 armory.

Mark Twain, who lived in the nearby Mark Twain House, visited Colt's armory in 1868 and described it thus: "It comprises a great range of tall brick buildings, and on every floor is a dense wilderness of strange iron machines… a tangled forest of rods, bars, pulleys, wheels, and all the imaginable and unimaginable forms of mechanism… It must have required more brains to invent all those things than would serve to stock 50 Senates like ours."

Today the factory complex includes: the Forge Shop and the Foundry (from the original 1855 factory); the East Armory with its distinctive blue onion dome, rebuilt in 1867; the South and North Armories (1921), the Machine Shop, Warehouse, Power Plant, and Garage, built in 1916 to accommodate World War I production; and the World War II Office Building (1942).