His father was a distinguished British Army intelligence officer during the First World War who became a businessman.

Woolrych was descended from an old Shropshire gentry family, and was related to Sir Thomas Wolryche, 1st Baronet, Royalist governor of Bridgnorth during the English Civil War.

Woolrych was educated at Westminster School, but owing to the family's financial distress during the Great Depression he did not proceed to university.

After the war, Woolrych was decommissioned with the rank of Captain, and attended Pembroke College, Oxford on an ex-serviceman's educational grant; he originally planned to read English, but owing to the college's lack of a tutor read History instead.

He remained there until 1964 when he became Professor of History at the newly founded University of Lancaster.