Born in Houston, his family moved to Portland, where he attended Jefferson High School, starring as a two-way football player, in track and field and basketball.
The Democrats 1958 team is regarded as one of the greatest in Oregon prep history, in that season he broke the Portland Interscholastic League record with 24 touchdowns and was part of a backfield that included him at fullback, quarterback Terry Baker (Heisman Trophy winner at Oregon State University), his younger brother Mel Renfro at halfback (future Pro Football Hall of Famer) and halfback Mickey Hergert (one of the leading ground gainers in the nation at Lewis & Clark College).
Renfro for his time had an excellent size/speed ratio, being 6–3 feet, around 210 pounds and could run the 100 yard in 9.6 seconds.
He transferred to Portland State University in 1962, reuniting with his former high school coach Tom DeSylvia.
[3] Raye married Virginia Smith on October 21, 1972 in Portland, Oregon, and adopted her three sons, Edward, Brent, and Scott.