Among his brothers were James, who would become a Member of the Western Australian Legislative Council, and John Nicol Drummond, first Inspector of the Native Police.
In 1829 the Drummond family emigrated to the Swan River Colony in what is now Western Australia, arriving on board the Parmelia on 1 June.
Drummond spent much of his early life helping his father and brothers run their farm at their grant on the Swan River.
The expedition, which included James Drummond Snr and Samuel Pole Phillips under the command of Captain John Scully, discovered the vast tract of open pastoral land that is now known as the Victoria Plains.
Drummond and his father began planning to make their entire living from collecting, discussing going to South Australia or India, but nothing came of it.