He was the third son of General Frederick William Buller of Pelynt and Lanreath in Cornwall, who had himself served with distinction in the 57th Foot in the Netherlands and the West Indies.

The first twenty-five years of his career in the army were spent in a time of peace, and his marriage with Henrietta, daughter of General Sir John Macdonald, Adjutant-General to the Forces, helped his rapid promotion.

In September 1847 he was appointed second in command to Sir George Berkeley in the campaign in the Amatola mountains, in which his battalion chased Sandilli so hotly that the chief surrendered to Buller on 19 October.

[1] In October 1853, Buller's battalion was again ordered home, and in spite of Sir George Cathcart's entreaties that he would remain as a brigadier at the Cape, he insisted on accompanying it.

When it was decided to send an expeditionary army to the East in 1854, Buller was appointed brigadier-general, and took the command of the 2nd Brigade of the Light Division, consisting of the 19th, 88th, and 77th Regiments.