[1][5] The church was one of the early small parishes begun by English families that settled in Riceville (now Navesink), New Jersey.

He, John Henry Stevens, and other members of their family and friends were the leaders in the formation of a congregation and the foundation of the parish of "All Saints' Memorial Church in the Highlands of Navesink."

All Saints grew and added three buildings to the complex: the parish house in 1865, the rectory in 1869, and the carriage sheds at the turn of the century.

The influence of St. James is seen in a number of these small parish churches like All Saints'--the simplicity, dignity, and simple stone masses without much ornament are typical of Upjohn's preference for what he called "truth" in Gothic Revival architecture, and with these small churches he had established the concept of taste and competence.

Moreover, the New York Ecclesiological Society aggressively maintained independence from its English counterpart, and in the articles which appeared in the New York Ecclesiologist, the American avoided as far as possible the complicated byways of high Ecclesiology, dealing instead with such matters as the honest use of materials, economy, and the need to maintain actual designs within the limits of local capability.