He was educated at home and at a private school in Maryland, before attending the College of William and Mary in Williamsburg, Virginia where he graduated with high honors in 1771.

He returned to the Colony of Virginia, and was serving as an instructor at William and Mary as the hostilities which led to the American Revolutionary War broke out.

The same year, Loyalist sympathies of the college's president, the Reverend John Camm (who had been the initial litigant in the Parson's Cause case 1758–1764), brought about Madison's removal from the faculty.

But later, Madison became the eighth president of the College of William and Mary in October 1777, the first after the outbreak of the American Revolutionary War.

The college's founder, Reverend Dr. James Blair had arranged financing for that purpose using dedicated income from England which was interrupted by the Revolutionary War.

The Reverend Dr. Madison also presided over the first convention of the newly formed Diocese of Virginia of the Episcopal Church in the United States of America in 1785 and was consecrated as a bishop on September 19, 1790, in the Lambeth Palace in London.