Built in 1908-09, it is one of the small community's most architecturally distinctive buildings, designed in the Romanesque Revival style by a local architect.

A semicircular section projects from the southern part of the eastern facade, its semi-conical roofline joining that of the main roof.

The interior is arranged similarly to other 19th-century libraries, with a central circulation desk, a reading room in the semicircular section, and the stack area to the rear.

Daniel Caswell, a Harrison native living in Texas, donated $1,000 to a building fund, conditioned on it being named for him.

There is a distinct stylistic resemblance between the library and Caswell's house in Austin, Texas, suggesting the benefactor may have had an influence on the design.