James Lillywhite (23 February 1842 – 25 October 1929) was an English Test cricketer and an umpire.

Lillywhite is termed "junior" in sources to differentiate between him and his cousin James senior.

He was one of the organisers of Arthur Shrewsbury's team to Australia in 1884–85 but, in spite of his experience, the Australian captain Billy Murdoch refused to allow him to umpire the first ever Test match at Adelaide.

However, along with Ted Elliott, he umpired in the second Test of that series, when the entire Australian team refused to play unless they received fifty per cent of gate takings.

Lillywhite's other match as umpire was the drawn fourth Test between England and Australia at Old Trafford in 1899.