He was born in San Jose, California, and trained with the Santa Clara Swim Club under coach George Haines.

[1] As a member of the Santa Clara Swim Club, he won three Amateur Athletic Union (AAU) United States national championships.

Ivey won the 200-yard backstroke at the NCAA Men's Swimming and Diving Championships with a time of 1:52.77 in 1970,[2][3] and graduated from Long Beach State in 1972.

[13] Ivey denied the charges of misconduct, saying "I was told that putting my arm around a girl and using foul language was deemed reason enough [for the University of Florida to fire him].

[13] His Florida women's swimmers issued a unanimous statement supporting him,[13] and stated publicly they did not complain about nor witness any inappropriate behavior.

Ivey, a two-time Olympic medalist who coached at Concord Pleasant Hill Swim Club at the time, allegedly went into her hotel room and had unwanted sex with her during the 1983 U.S. Championships in Indianapolis on a trip chaperoned by King.