Wither was born in 1822 at Manydown Park, a manor in Wootton St Lawrence, Hampshire, England.

[1] His father was Harris Bigg-Wither (1781–1838), who was Lord of the Manor at Manydown from 1813 to 1833[2] and is famous for having proposed to Jane Austen.

When William Wither V (1733–1789) died, he had no heir, and a cousin with the surname Bigg—the Reverend Lovelace Bigg (1741–1813)—became his successor.

[7] The Wither Hills south of Blenheim, which were part of his sheep run, are named for him.

[5] Wither was one of the original nine trustees of Nelson College; other notable foundation trustees were Charles Elliott, David Monro, John Barnicoat, William Wells, and Alfred Domett.