He joined the music department of the 51st Defense Battalion, which was led by Bobby Troup, and Wiltshire was able to hear and play with such musicians as Lionel Hampton, Duke Ellington, and Louis Jordan.
In 1951 he began working in A&R for Prestige Records, and his band backed King Pleasure on his 1952 hit "Moody's Mood for Love", an early example of vocalese, the addition of lyrics to a jazz improvisation.
The same year, Wiltshire led the band – including himself on piano, Percy Heath on bass, and Art Blakey on drums – behind Annie Ross on her successful original recording of "Twisted".
During the late 1950s, he was increasingly seen as an orchestrator and arranger, and worked on a freelance basis with various musicians including the Fidelitys, Nappy Brown, and Marie Knight.
In the early and mid 1960s, he collaborated with top R&B producers Leiber & Stoller and Bert Berns, working with acts such as Chuck Jackson, Derrick Harriott, The Exciters, The Isley Brothers – for whom he arranged their hit, "Twist and Shout" – The Shirelles, and Wilson Pickett.