Rebhorn appeared in numerous films, portraying Los Alamos Doctor in Silkwood (1983), Drunk Businessman in Cat's Eye, Richard in Heart of Midnight (1988), Prosecutor in Desperate Hours (1990), Dr. Sultan in Regarding Henry (1991), George Wilbur in My Cousin Vinny, Dr. McElwaine in Basic Instinct, and FBI Agent Flynn in White Sands and Mr. Trask in Scent of a Woman (all in 1992), Ellard Muscatine in Lorenzo's Oil, and Bill Norwalk in Carlito's Way (all in 1993), Howard Schaeffer in Guarding Tess and Mando, The Thin Man in I Love Trouble (both in 1994), Captain Tyler in White Squall, John Merino in Up Close & Personal, and Albert Nimziki in Independence Day (all in 1996), Charlie Reynolds in My Fellow Americans and The Game (both in 1997), Alvin Hooks in Snow Falling on Cedars and Herbert Greenleaf in The Talented Mr. Ripley (both in 1999), and President Signoff in The Adventures of Rocky and Bullwinkle and Dr. Larry Banks in Meet the Parents (both in 2000); as well as recurring roles in TV series, such as his portrayal of Charles Garnett/ Dr. Horace Garrison/Albert Lawrence Cheney in Law & Order, Captain Elchisak in Third Watch, Bertram Webster in The Book of Daniel, Reese Hughes in White Collar, Carl Franklin in Big Lake, and Frank Mathison in Homeland.

Richard S. Huffman was one of his instructors and directed him in a lead role in Aristophanes' Lysistrata as well as Molière's Tricks of Scapin.

Rebhorn played Peter Latham in Forty Carats at the GasLight Dinner Theatre in Salt Lake City in the 1970s.

He delivered equally notable performances in a variety of other roles, including that of a brutal serial killer on NBC's Law & Order (he would later return to the show in the recurring role of defense attorney Charles Garnett), Ellard Muscatine in Lorenzo's Oil (1992), Fred Waters in Blank Check (1994), Clyde Frost, the father of famed bullrider Lane Frost, in 8 Seconds (1994), Lt. Tyler in White Squall (1996), and a shipping magnate in The Talented Mr. Ripley (1999).

In 2003, he appeared in the Chris Rock film Head of State, as Democratic Senator Bill Arnot.

His role in the short-lived and controversial NBC drama The Book of Daniel cast him as the father of the title character.

From 2009 to 2013, Rebhorn played the recurring role of Reese Hughes, the FBI white-collar division chief, on White Collar.

In 2012, he played the role of Archbishop McGovern in "Leap of Faith",[7] the 13th episode of the second season of the CBS police procedural drama Blue Bloods.

His stage career included seven Broadway productions, as well as numerous appearances with New York City's Roundabout Theatre Company.