In 1939, he attended the University of Alabama but interrupted his studies to serve a tour of duty in World War II with the 44th Bombardment Group (Heavy) of the Eighth Air Force as a second lieutenant and navigator on B-24 Liberator bomber aircraft.

[1] After the war, he played semi-professional football and boxed while earning his master's degree in English and comparative drama from Columbia University.

Through the 1970s, Sweet took roles in films such as Colossus: The Forbin Project (1970), The Out-of-Towners (1970), The New Centurions (1972), Fear Is the Key (1972), Sisters (1972), Cops and Robbers (1973), The Lords of Flatbush (1974), Amazing Grace (1974), The Bad News Bears in Breaking Training (1977), Which Way Is Up?

He had a notable role as FBI director J. Edgar Hoover in the 1978 television miniseries King, based on the life of Martin Luther King Jr. As the 1980s began, Sweet worked steadily in films such as Below the Belt (1980) and Reds (1981), the made-for-television movie Gideon's Trumpet (1980) and television series such as Hill Street Blues and Hart to Hart.

Sweet's best-known television character was police chief and father Carl Kanisky, employer of housekeeper Nell Carter, on the sitcom Gimme a Break!.

Sweet underwent unsuccessful abdominal surgery in the summer of 1984 and was diagnosed with stomach cancer during the fourth season of Gimme a Break, but he continued to work.