Born in New York City, Stoller studied with drum teacher Henry Adler and launched his career touring and recording with swing era big bands led by Benny Goodman, Tommy Dorsey, Harry James, and Charlie Barnet.
He backed singers including Billie Holiday, Mel Tormé, and Frank Sinatra on some of their major recordings.
[1] Stoller also recorded with Art Tatum, Roy Eldridge, Oscar Peterson, Coleman Hawkins, Ben Webster (see Coleman Hawkins Encounters Ben Webster), Benny Carter, Herb Ellis, and Erroll Garner among many other jazz musicians.
In the 1950s, Stoller settled in the Los Angeles area, where he became respected for his work in the Hollywood studios, lasting for several decades.
On September 23, 1951, Stoller married Mary Hatcher, an American singer and actress, in Westwood, California.