Born at Edgbaston, Birmingham, on 13 February 1840, he was the son of Charles Bailey, by his wife Mary Elizabeth, daughter of John Eglington of Ashbourne.

He completed his education by attending evening classes at Owens College, Manchester, learned Pitman's shorthand, and contributed articles to short-hand manuscript or lithographed magazines.

[2] In 1881, Bailey started a monthly antiquarian magazine, the Palatine Note-Book, which ran for just over four years and ceased with the forty-ninth number in 1885.

[3] Bailey collected works on stenography with a view to writing its history, and built up a valuable library of antiquarian and general literature.

[2] In 1874, as the fruit of his researches, Bailey published a life of Thomas Fuller, which gained him admission to the Society of Antiquaries of London, and he was a contributor to the earliest volumes of the Dictionary of National Biography.