The synagogue, owned by the Congregation Adath Israel, is located in the former H. H. Baxter Memorial Library building, a Richardsonian Romanesque structure that is listed on the National Register of Historic Places.

The RJC building is a cruciform structure, built out of grey marble quarried in West Rutland and laid in irregular courses.

Prominent features include the arched entrance portico at the corner of Grove and Library Streets, and the multi-stage square tower that has an open top stage, each side featuring a pair of round-arched openings separated by a round column.

The building was constructed as a library in 1889 by Mary E. Roberts Baxter in honor of her husband, Horace Henry Baxter, a New York City financier and part-owner of the Rutland Marble Company.

[2] In 1927, the building was purchased by Congregation Adath Israel, which had been formed in Rutland in 1911.