Hunter-Blair was born in 1926 in Savoy House, Ayr, Scotland, the youngest son of Sir James Hunter Blair, 7th Baronet and his wife Jean Galloway McIntyre.
Following demobilization in 1948, he attended Balliol College, Oxford, graduating with a degree in history and embarking initially on a career in merchant banking in London.
Sir James Hunter Blair possessed an extensive knowledge of architecture, furniture, Scottish painting, music, angling, trees and garden design.
He was devoted to opera and also helped stage revues at the Gaiety Theatre in Ayr, including a memorable performance in 1956 – playing Julius Caesar on roller skates.
Whenever the Open Championships came to nearby Troon or Turnberry, up to 80 guests would find accommodation in the mansion, Milton, and in seven holiday cottages, converted from outbuildings.