He was born in Quebec around 1784, the son of Thomas Fraser, and grew up in Edwardsburgh Township in Upper Canada.

He served as lieutenant in the Dundas County Militia during the War of 1812, becoming captain in 1813 and raising his own Troop of Provincial Light Dragoons, and fought at the Battle of Crysler's Farm.

In 1818, he was fined for an assault on Robert Gourlay who was holding meetings in the eastern part of Upper Canada.

In 1839, Fraser triggered some friction between Canada and the United States when he seized an American schooner for failing to declare a cannon to customs.

By 1840, several banks had secured judgements against him for unpaid loans and the government was pressing him for customs duties that he had collected but not provided.