[4] Oliver entered Phillips Academy, Andover in 1811 and Harvard College in the fall of 1814, however because Harvard was becoming overly Unitarian in its views[5] and also more expensive,[5] Oliver left Harvard after his sophomore year and transferred to Dartmouth College entering Dartmouth's Junior class in the fall of 1816.

[5] Oliver also received a degree Ad eundem from Harvard.

At the age of 25, in 1825, Oliver joined the Unitarian church.

The same year, he married Sally Cook, and they had seven children.

[6] In 1844 Oliver was appointed the Adjutant General of Massachusetts by Governor George N.