[1][2] He married Margaretta Taylor Amyatt in Walcot on 5 October 1815;[3] Brown was recorded as a lieutenant colonel in the 23rd Regiment of Foot at the time of his marriage.

[14][15] During his time as commandant of the British Army's logistics base in Belém, Brown had taken note of the plight of the children of servicemen on campaign and set up a school for these camp followers to bring them closer to the benefits of even a rudimentary education.

[18] He became commandant of the Royal Military Asylum[19][20] on 15 December 1843 as a brevet colonel[21] and retired from that post as a major general on 1 April 1852.

[22] By the time Colonel Brown took command of the Royal Military Asylum, the institution had already developed as one of the best free education provisions for the sons and daughters of servicemen, based on the Madras School system.

[23] In the case of the Royal Military Asylum children were placed as orphans or as in dire need of stable schooling based on recommendations from the fathers' regiments.