Rose was born in Washington, D.C. His parents were Jewish immigrants, his father from Bragin, Belarus, and his mother from Kyiv, Ukraine.

After completing his studies at Philadelphia's Curtis Institute of Music at age 20, he joined Arturo Toscanini's NBC Symphony Orchestra, and almost immediately became associate principal.

He made many recordings as a soloist after 1951, including concertos with conductors such as Leonard Bernstein, Eugene Ormandy, George Szell and Bruno Walter among others.

His pupils include Lori Singer, Raymond Davis, Desmond Hoebig, Peter Stumpf, Fred Sherry, Christopher von Baeyer, Myung-wha Chung, Patrick Sohn, Thomas Demenga, Stephen Kates, Lynn Harrell, Yehuda Hanani, Hans Jørgen Jensen, Steven Honigberg, Eric Kim, Roger Drinkall, Robert deMaine, Bruce Uchimura, Donald Whitton, Yo-Yo Ma, Ronald Leonard, Steven Pologe, Sara Sant'Ambrogio, Matt Haimovitz, Mats Lidström, Richard Hirschl, John Sant’Ambrogio, and Marijane Carr Siegal.

Ararat Cemetery in Farmingdale, New York, next to the grave of his first wife, Minnie Knopow Rose, who died in 1964.