He was born in Alexandria, Virginia, the son of the rector of Christ Church, William Holland Wilmer and his wife Marion Hannah Cox, who died in childbirth when Richard was six.

Raised in part by his stepmother (his eldest brother dying in Mississippi shortly after his father), Richard Wilmer worked as a teenager to support his family.

However, Bishops James Otey of Tennessee and Thomas Atkinson of North Carolina refused to participate in the necessary consecration ceremony before the PECCA held a convention and established its form of government.

He could not attend the first General Convention after the war because he was under house arrest for instructing his clergy to omit prayers for the President of the United States as a protest against military occupation.

Although another Virginia priest, Henry Melville Jackson, assisted Wilmer in his last years, the diocese elected Robert Woodward Barnwell, a native of Selma, Alabama, as his successor.