By 1558 their father, Lord Montagu, had married secondly Magdalen Dacre, by whom he had three further sons, George, Thomas and Henry, and three daughters, Elizabeth, Mabel and Jane.

[citation needed] On 19 February 1565/66, at the age of thirteen, Browne's twin sister, Mary, had married, from their father's house in London, Henry Wriothesley, 2nd Earl of Southampton.

When, in, 1580 it was reported to him that she had been seen at Dogmersfield with Donsame, he forever banished her his 'board and presence', forcing her to live at one of his Hampshire estates under close surveillance.

The Countess defended herself with spirit in a long letter to her father, Viscount Montague, on 21 March 1580, denying adultery and accusing one of the Earl's servants, Thomas Dymock, of having been the cause of the contention between herself and her husband.

An entry in the register of the Privy Council records that one of the 2nd Earl of Southampton's servants had been committed to the Marshalsea on 23 February 1580 "for certain misdemeanors by him used against Mr Anthony Brown, the eldest sonne of the Lord Montacute".

Shortly after Queen Elizabeth's visit to his father's house at Cowdray, Browne was denounced by the informer Robert Hammond to Lord Burghley for associating with Catholics whom the regime considered suspect.

The Jesuit John Curry was reported by Viscount Montague's chaplain, Robert Gray, to have been residing at Riverbank House[8] in Cowdray Park,[9] and Thomas Simpson, who later preached Browne's funeral sermon, and had been ordained a priest in the early 1580s by the Cardinal of Guise, may also have been a visitor there.