He never had a lot of contact with his much older brother Michael Rossi and grew up in New York under the care of his sister Lisa.

[1] In his teens, he spent most of his time hanging out with a bad crowd and was often put in juvenile's prison for theft and other minor criminal acts.

[1] Offscreen, in the late 1960s, Joe had a short-lived relationship with Jill Smith, who suddenly left him.

However, it is later revealed he fled to town to avoid the revenge of a hoodlum friend in New York City.

[2] It later turns out Joe and Jill together made up a story on their baby to scheme the residents of Peyton Place.