In the mid-19th century, it emerged as a stopover for farmers traveling to the Whitby port, and gradually evolved into a village.

[3] The early settlers of Audley mainly included the English, the Scots, the Irish migrants escaping the Great Famine, and the Quakers from New England whose advocacy for pacifism had invoked hostility there.

[2] Audley developed as a stopover for people traveling to the port of Whitby, and as a commercial centre for farmers from the surrounding area.

Its owner Abraham Brown also ran a bar, and kept a menagerie of caged wild animals near the hotel.

[3] On Saturdays, men from the surrounding areas converged at Audley for drinking and sports, and often engaged in unruly behavior including fights.

[4] The local Methodist Church was established to discourage this behavior, and partnered with the Temperance Society to campaign against the consumption of alcohol.

[6] The village also had two blacksmiths, a cobbler, a tanner, a butcher, a wagon-dealer, a paint shop, and a general store.

[3] In the late 19th century, a fall in the demand for wheat led to economic decline in the Pickering Township.

[7] The Audley Public School was established in 1840, in a one-room log building on Lot 10 of Concession Road 3.

Nattress and fellow preacher Benjamin Haigh held several meetings in the homes of local families that supported the idea.

In the 1860s, James Madill and John Lawrence built a frame building for the church, on land donated by William and Jane Bell, with funds from several local residents.

The Post Office closed at the start of World War I in 1914, and was replaced by rural letter carriers.

[14] In 1926, the local residents established the Audley Community Club, which organized events such as plays, box socials, dances, baseball games, and picnics.