He was a member of the Scottish National War Memorial committee formed in 1918 by the then Secretary of State for Scotland, the Liberal MP Robert Munro, which was tasked with forming a permanent memorial for World War I at Edinburgh Castle.

In 1927 he criticised the decision to include Walter Thomas Monnington's painting Parliamentary Union of England and Scotland, 1707[3] in the Palace of Westminster which he said in a debate should "get rid of this disgraceful picture and put something in its place which would be more true to history than it is."

"[4] In his authorised biography, Bill Shankly said that Brown won the election in South Ayrshire as the local voters "wouldn't vote Conservative" but that he was "too mild" and not enough of a "keen socialist" for some of the more militant Labour members.

He was granted the Freedom of the Royal Burgh of Ayr in 1930, and of Girvan in 1931 and was awarded an Honorary LLD by the University of Glasgow in 1931.

According to newspapers, during his time as Lord High Commissioner he developed a rapport with George V. James and his wife had five children; two died before adulthood and one son was killed in the First World War.

James Brown died on 21 March 1939 in an Ayr nursing home at the age of 76 from heart failure, with his children at his bedside.