His other film roles include in Buffalo Bill and the Indians, or Sitting Bull's History Lesson (1976), Kafka (1991), The Music of Chance (1993), The Fantasticks (2000), and Dancer in the Dark (2000).

He changed his last name from Katz to Grey early in his career due to the stigma associated with having a surname with an obvious ethnicity attached.

He returned to Broadway in The Littlest Revue in 1956 and acted as a replacement in Neil Simon's Come Blow Your Horn in 1961 and the musicals Stop the World – I Want to Get Off in 1962, and Half a Sixpence in 1965.

He then took on roles in the late 1950s and early 1960s, Grey appeared in several TV westerns including Maverick (1959), Bronco (1960) and Lawman (3 times in 1960 and 1961).

Grey gained his breakthrough performance originating the role of the Master of ceremonies in the Broadway musical Cabaret by John Kander and Fred Ebb in 1966.

Fosse, who was hired to direct the film version because Harold Prince was unavailable, wanted to recast the MC role, but the studio insisted on Grey.

Grey's character Chiun was popular for the lines "Meat of cow kills", and "You move like a pregnant yak", from the movie.

He then acted in Steven Soderbergh's mystery thriller Kafka (1991), starring Jeremy Irons, Theresa Russell and Ian Holm.

[15] That same year, Grey also appeared in the American Repertory Theater's production of When We Dead Awaken at the São Paulo Biennial.

The following year he starred in the Philip Haas drama film The Music of Chance (1993) alongside James Spader, Mandy Patinkin, M. Emmet Walsh, and Charles Durning.

Later that year he starred in New York Stage and Film's production of John Patrick Shanley's A Fool and Her Fortune and received a nomination for the Primetime Emmy Award for Outstanding Guest Actor in a Comedy Series at the 45th Primetime Emmy Awards for his recurring role as Jacob Prossman on the television series Brooklyn Bridge.

In 1995, he made a guest appearance on the Star Trek: Voyager episode "Resistance" as Caylem, an aging rebel seeking to free his (deceased) wife from prison.

He played Milton Winters, a wealthy, paroled ex-convict on Law & Order: Criminal Intent (episode "Cuba Libre", 2003).

He also appeared on the shows House and Brothers & Sisters (2007), on the latter of which he played the role of Dr. Bar-Shalom, Sarah Walker (Rachel Griffiths) and Joe Whedon (John Pyper-Ferguson)'s marriage counselor.

He appeared as Dr. Singer, Dr. Izzie Stevens (Katherine Heigl)'s high school teacher who needs treatment for dementia in Grey's Anatomy (2009).

Grey took over the role from Robert Morse who previously played the Wizard in the San Francisco tryout run at the Curran Theatre.

He returned to Broadway in the 2016 revival of the Anton Chekhov play The Cherry Orchard starring opposite Diane Lane, and Chuck Cooper.

In 2018, Grey directed a Yiddish-language production of Fiddler on the Roof, which originated at the National Yiddish Theatre Folksbiene, then transferred to Stage 42 Off-Broadway.

In 2022 he acted as Morgan Bote, a recurring character in the FX drama series The Old Man starring Jeff Bridges and John Lithgow.