The family or the sheriff believed he was poisoned by slaves, and three were charged in the case and convicted by justices of the Commission of Peace.

[1] Unusually, only one slave was executed; Dido and Turk, owned by the widow Frances Taylor Madison, were returned to her to serve as laborers after being punished by whipping.

His widowed mother never remarried, which was unusual at that time for a woman her age; she had extensive Taylor family in the county.

Acquiring more land, he eventually owned 5,000 acres, making him the largest landowner in Orange County.

They had twelve children: During the American Revolution, Madison served as chairman of the Orange County Committee of Safety.