He was the second son of Michael Millingen, a Dutch merchant originally from Rotterdam and then from Batavia in the western Netherlands, emigrated to England and settled in Queen's Square, Westminster (now Bloomsbury, Camden).
James was educated at Westminster School, and attracted the attention of his father's friend and neighbour, Clayton Mordaunt Cracherode, who encouraged him to study numismatics.
Late in 1792 Millingen was arrested as a British subject by a decree of the National Convention, and confined first in the prison of the Madelonettes, then in that of the Luxembourg, and finally in the Collège des Écossais, where he remained until released after the Thermidorian Reaction in July 1794.
[1] On obtaining his liberty Millingen settled in Calais, but subsequently became a partner in the banking house of Sir Robert Smith & Co. in the Rue Céruti, Paris.
For some time he lived at Rome and at Naples, where he made the acquaintance of Lady Blessington, but latterly settled at Florence, paying occasional visits to Paris and London.