They had eight children, including three daughters and one son who predeceased their parents: Mary Anne at Petersburg, Virginia in 1862, Julia at her uncle's historic house Eldon in Pittsylvania County, Virginia (later known as the estate of Claude A. Swanson), Jane Eliza at Hopkinsville, Kentucky circa 1872 and Llewellyn Fairfax Whittle circa 1880.

He then moved to Louisville, Kentucky, where he served at St. Paul's Church through the American Civil War, although his family returned to Virginia.

In 1897, Robert Atkinson Gibson was elected as bishop Whittle's Coadjutor with right of succession (the previous assistants having moved to the Diocese of Southern Virginia or died) and soon consecrated.

While Bishop Whittle in 1878 had established the Bishop Payne Divinity School in Petersburg, Virginia to prepare African-Americans for ordination (which ultimately merged with VTS) and ordained George Freeman Bragg as deacon in 1887 and as priest the following year, in 1899 the diocesan council approved restricting Bragg's voice (and that of other non-white clergy) to the Convocation of the Missionary Jurisdiction.

Bishop Whittle also was active in sending missionaries to Brazil, and one of his last acts was signing a petition of clergymen opposing the war in the Philippines.

His Greek Revival house in Richmond, 21 West Clay Street, lies in the Historic Jackson Ward neighborhood.