His father Edward Lawrence Swem (1838–1918) (a gallery owner in Belle Plaine and later Cedar Rapids), was born in Indiana.

[2] Whilst attending the local high school, young Earl began working at the Iowa Masonic Library.

[3] They had one son, Earl G. Swem, Jr., who later attended the College of William and Mary and was a member of a secret society there.

Around 1903 he moved to Washington, D.C., and began working at the Library of Congress, becoming chief of the cataloging division of the Copyright Office.

After his retirement from William and Mary in 1944, Swem continued to write short articles, as well as edit books and manuscripts on Virginia history and serve as librarian emeritus.