He was the second son of Charles Bruce, 5th Earl of Elgin and his wife Martha Whyte, governess to Princess Charlotte of Wales.

[15] On 11 March 1799, shortly before setting off for Constantinople, Elgin married Mary, daughter and heiress of William Hamilton Nisbet, of Dirleton.

As ambassador to the Sublime Porte he showed considerable skill and energy in fulfilling a difficult mission, the extension of British influence during the conflict between the Ottoman Empire and France.

[18] Following discussions with the diplomat and archaeologist Sir William Hamilton, Elgin decided he would engage, at his own expense, a team of artists and architects to produce plaster casts and detailed drawings of ancient Greek buildings, sculptures and artefacts.

Elgin stated that about the middle of the summer of 1801, he had received a firman from the Sublime Porte which allowed his agents not only to "fix scaffolding round the ancient Temple of the Idols [the Parthenon], and to mould the ornamental sculpture and visible figures thereon in plaster and gypsum," but also "to take away any pieces of stone with old inscriptions or figures thereon".

The decision to remove marbles attached to structures was made on the spot by Philip Hunt, Elgin's chaplain and one of his representatives in Athens.

[20] Elgin's agents removed about half of the Parthenon frieze, fifteen metopes, and seventeen pedimental sculpture fragments, in addition to a caryatid and a column from the Erechtheion,[23] sculptured slabs from the Athenian temple of Nike Apteros, and various antiquities from Attica and other districts of Hellas.

[14] Part of the Elgin collection was prepared for embarkation for Britain in 1803, and considerable difficulties were encountered at every stage of its transit.

On Elgin's departure from the Ottoman Empire in 1803, he withdrew all his artists from Athens with the exception of Lusieri, who remained to direct the excavations, which were still carried on, though on a much reduced scale.

Additions continued to be made to the Elgin collections, and as late as 1812, eighty fresh cases of antiquities arrived in England.

[14] Elgin's procurement of the marbles was supported by some, including Goethe,[24] and censured by others in Britain as vandalism, most famously Lord Byron,[25] who wrote the following lines[26] Dull is the eye that will not weep to see Thy walls defaced, thy mouldering shrines removed By British hands, which it had best behoved To guard those relics ne'er to be restored.

In October 1805, the French gave Lady Elgin, who was pregnant, permission to return to England on compassionate grounds.

Elgin himself was finally allowed to leave France in June 1806 following a direct appeal to Napoleon by British Prime Minister Lord Grenville.

[32] Soon after returning to Britain, Elgin discovered that his wife was having an affair with Robert Ferguson, one of his oldest friends.

In December 1807, Elgin successfully sued Ferguson for seduction in the English courts and was awarded £10,000 in damages.