His mother was Anna Maria, daughter of Richard Stout, judge and member of the governor's council on Cape Breton Island.

In 1844 he married Catherine Prior Hall, with whom he had six children, including Georgiana Poulett Harris (Mrs. Ingle, 1845–1919), first headmistress of Christchurch Girls' High School in New Zealand.

After his wife's death in June 1856, Harris went to Tasmania to become headmaster of the Hobart high school, and filled the position with much ability, inspiring both respect and affection from his pupils.

He was one of the original members of the council of education founded in 1859, and long advocated the establishment of the University of Tasmania.

Louisa's younger twin, Harriet Lily, died from Tubercular Peritonitis in 1897, aged 23, having been a teacher at the Ladies' Grammar School and Kindergarten in Davey Street, Hobart that was run by her sisters.