Born in Newark, New Jersey, Carlesimo graduated from Saint Benedict's Preparatory School, then attended Fordham University in The Bronx, where he played football alongside Vince Lombardi.
As a member of the Metropolitan Intercollegiate Basketball Association (MIBA), Fordham was part of the committee that ran the National Invitation Tournament.
Carlesimo proposed moving the early round games to campus sites, and having only the final four teams play at Madison Square Garden in New York City.
Carlesimo was known as a humorous speaker, having appeared at countless sports-related dinners and conferences, and was once a guest on The Tonight Show Starring Johnny Carson.
He was married to Lucy Rogan and had ten children, including P. J. Carlesimo, the eldest, who was a longtime coach at Seton Hall University and in the National Basketball Association (NBA).