Ryder's lifelong connection with John Henry Newman and the Oratory began as a private pupil, when he was about twelve years old.
The only interruption was a year at the English College at Rome and a few months at the Catholic University of Ireland in Dublin, of which Newman was rector, before he began in December, 1856, his Oratorian novitiate.
After Newman's death he was elected superior of the Birmingham Oratory and held this office till his health gave way.
He was the last survivor of "my dearest brothers of this House, the Priests of the Birmingham Oratory" to whom Newman dedicated his Apologia Pro Vita Sua.
He was the eldest son of George Dudley Ryder, one of the numerous clergymen of the Church of England who followed in the steps of Newman, and Sophia Lucy, daughter of John Sargent.