Along with other families, such as the Andersons, Jordans, Cabells and Childresses, both sides of the river were settled by people moving up from the lower James to plant new lands with tobacco and other crops in the 1730s and 1740s.

The area was also known for its limestone and iron ore deposits and a forge of some sort was established in the settlement from a very early date.

The original community consisted of several large and medium plantations, as well as a landing for bateaux poling up and down the river.

Because of its large, secure brick warehouse, its foundry, and other secure brick buildings, as well as its central location on the James River, with roads heading both east-west and north-south, it became one of the storage depots for the Virginia Militia and several letters of the time from the major in command at Irving's Store talk of difficulties in supplying the militia with armaments because of the low level of the river and lack of boats to transport goods.

It is also mentioned that British cavalry under Colonel Tarleton tried to make a raid up the river from Richmond to destroy the arms depot.

Within ten years of the canal coming to the town, local investors set up the Howardsville and Rockfish Gap Turnpike Company and during the 1840s cut a road from the river over the Blue Ridge Mountains and across the Shenandoah Valley south of Staunton all the way to the Appalachians.

Because of its industry and location on the canal, Union cavalry under Sheridan, which including a column led by George Custer, raided the Howardsville-Scottsville area in the spring of 1864, burning bridges, the foundry, the mills and several the houses.

It burned and smashed the lock gates and aqueduct, destroyed canal boats carrying food and military supplies, raided the bank - though its gold had been secreted away just hours before the Union troops arrived – and Sheridan set up house in Monticola while his men pillaged the surrounding area.

Triumphant, they eventually headed east down the river, to then turn northeast and circle Richmond to rejoin Grant's army.After the Civil War, the community struggled to regain its former glory.

The oldest, most historic homes, which were sheltered from hurricanes Camille and Agnes, remain, having been restored by their present owners.