Born at Stone Hill plantation about a mile east of the county seat to Hugh Rogers and his wife, the former Mary Coombs, he was one of eleven children, seven of whom survived to adulthood.

Rogers also operated a store at Madison and Washington Streets (which remains today as a contributing property to the Middleburg Historic District).

In 1836, the Baltimore and Ohio Railroad was completed, and by the 1840s, the wagon route trade had diminished significantly, to the detriment of merchants such as Asa Rogers.

In 1864, General Grant ordered dozens of Loudoun citizens, including Rogers, arrested and imprisoned, supposedly because they were subject to Confederate conscription.

Rogers was among 32 arrested and imprisoned in Washington D.C. in an attempt that Confederate authorities would free 26 Pennsylvania residents similarly carried off during the Gettysburg campaign.

Rogers refused to take the oath to be paroled, stating that he had a son and nephews in the Confederate army and if they came to his house, he would even attempt to prevent Union forces from capturing them.

[12] Both Rogers brothers had mortgages on their six large estates (Stone Hill, Mill Hill, Texas Farm, Oakham Farm, Dover, Ellendale, and Clifton), and Dover especially (owned by their brother William, who had been accused of fraud and self-dealing at the expense of the widow Hixon as early as the 1840s) was subject to lawsuits.