A Toronto native, Williams threw and batted right-handed, stood 5 feet 10 inches (1.78 m) tall and weighed 180 pounds (82 kg).

His most significant achievement was his tenure coaching first base with the Baltimore Orioles (1981–1987), a run which included their historic 1983 World Series win.

[1] Williams graduated from De La Salle College School of Toronto, where he played varsity baseball, hockey (as a right wing), football and lacrosse.

Williams met his first wife, Lorraine Elliott, while playing for the Pueblo Dodgers in 1949, with whom he had three children, Chris, Kirk and Julie.

In 1988, he married Carole Garrett of Baltimore, and continued his home-improvement work in the off season; their wedding date was inscribed on the front porch of their home in Joppa, Maryland.