[4] Costabile has appeared on Broadway in Manhattan Theatre Club's revival of Translations in 2007, the musical Titanic in 1997, and The Tempest in 1995.

For Caroline, or Change, the Drama Desk Award–winning musical by Tony Kushner and Jeanine Tesori (2003–2004), Costabile workshopped and performed in the original production and its Broadway transfer.

Costabile is best known for his recurring television roles on The Wire (as Thomas Klebanow, managing editor of The Baltimore Sun), Flight of the Conchords (as Doug), Damages (as Detective Rick Messer), Suits (as Daniel Hardman, former senior partner of the law firm of Pearson Hardman), Breaking Bad and Better Call Saul (as Gale Boetticher),[5] and the Showtime production Billions (as Mike "Wags" Wagner).

Costabile also appeared in season six of The Office, as a banker sent from Sabre to evaluate Dunder Mifflin, and in the House episode "Changes" as a limousine driver.

In 2011, he appeared in Lie to Me as Dr. Mitch Grandon, the head doctor of a psychiatric institution in the Season 3 episode "Funhouse".

Costabile appeared as Internal Affairs investigator Simon Boyd in the 2013 TV show Low Winter Sun.