Bailey was born near Fishkill in the Province of New York on October 12, 1758 where he attended the rural schools and studied law.

He was admitted to the bar in 1778 and commenced practice in Poughkeepsie, New York.

He also served in the State militia from 1786 until 1805 and attained the rank of brigadier general.

In April 1801, he was elected to the New York State Assembly, but vacated his seat before the State Legislature convened, because he was elected to the 7th United States Congress to fill the vacancy caused by the resignation of Thomas Tillotson, and served from December 7, 1801, to March 3, 1803.

He was interred at Dutch Burying Ground, Manhattan, New York; and was re-interred on January 8, 1864, at Poughkeepsie Rural Cemetery, Poughkeepsie, New York.