Sir James Caldwell, 4th Baronet, Count of Milan (c.1720 – February 1784) was an Anglo-Irish soldier and author.
He served as aide-de-camp to Charles Emmanuel III of Sardinia, and was employed chiefly in negotiating with the Kingdom of Great Britain.
He was made Count of Milan by Empress Maria Theresa in 1749, having inherited his father's baronetcy in 1744.
Unwilling to take the oath of allegiance to Austria, Caldwell declined the post of chamberlain to Maria Theresa and returned to Ireland in 1750.
He improved the estate at Castle Caldwell and gave land for the construction of a Roman Catholic church.