His older brother, William Wesley Woollen, was a banker, politician, and historian who wrote the book Biographical Sketches of Early Indiana.
His younger brother, Levin James Woollen, became a physician and a member of the Indiana Senate.
In 1856, Woollen moved to Vernon, where he read law and worked at the Jennings County Clerk's office.
After his brief stay in Vernon, Woollen moved again to Franklin, where he started a law firm in partnership with Jeptha D. New (later a U.S. Representative from Indiana).
He married his first wife, Harriet J. Williams (daughter of a Jackson County judge) in Brownstown in 1850.