He was born 5 September 1776 at Dunkirk, where his father, Andrew Frazer of the Royal Engineers, was Assistant Commissary in the demolition of the fortress.

The elder, Augustus Henry, was born in August 1810 and obtained a commission in the Royal Artillery He retired from service at the rank of second captain and died unmarried while travelling in Syria in July 1848.

The younger son, Andrew James, born in October 1812, obtained a Lieutenant's Commission in the Rifle Corps, from which he retired, and he died, unmarried at Ramsgate in July Educated at the High School, Edinburgh, where he was a contemporary of Lord Brougham,[1] Frazer joined the Royal Military Academy, at Woolwich, on 16 August 1790 as a gentleman cadet.

On 18 September 1793, he was gazetted a second lieutenant in the Royal Artillery, and at the end of the year joined the Army in Flanders, under command of the Duke of York.

In early 1794, he was promoted to First Lieutenant, and attached with 2 field guns to the 3rd Foot Guards (Grenadiers), where he served until the army returned to England in May 1795.

In 1803, Frazer received his rank of captain, and in 1807, embarked for South America to participate in General John Whitelock's expedition to Buenos Aires.

The Horse Artillery successfully dislodged the French from the woods surrounding Hougemont, without bringing the Allied troops under friendly fire.