His career was often controversial: in 1971 he was disciplined (overturned on appeal) after he gave a "V sign" to the judges following a near-perfect round which won him the British Show Jumping Derby for the second year in succession;[1] this act also earned him a 'tongue-in-cheek' part in an advertisement for Victory V sweets with the slogan 'They've got a kick like a mule!'

He wasn't fond of school and preferred to spend his time outdoors with animals, and bought his first jumping horse Farmer's Boy in 1954.

[4] Harvey will always be remembered for the immense contribution he made to the sport, not only as the ultimate showman, but as an Olympian who represented his country with fierce patriotism and whose desire never to be beaten drove him to win more than 50 grands prix, four Hickstead Derby titles, six championship medals and a swathe of prestigious trophies.

—Jennifer Donald, Horse & Hound 2020 [5]Smith won the John Player Trophy, the grand prix of Great Britain, seven times.

[5][7] In 1990, after retiring from show jumping competition, Harvey joined with wife and trainer Sue Smith, a former showjumper herself, to form a racing team at their Yorkshire base, Craiglands Farm.