[1][2] Born in Atlantic City, New Jersey, Albany studied piano as a child and, by 1943, was working on the West Coast in Benny Carter's orchestra.
[5] He continued for a few years afterward, and in 1957 recorded an album for Riverside with an unusual trio line-up with saxophonist Warne Marsh and Bob Whitlock on bass, omitting a drummer.
[3] His daughter Amy-Jo wrote a memoir about her father called Low Down: Junk, Jazz, and Other Fairy Tales from Childhood.
[8] Joe was also the biological father to Benjamin David Goldberg, who was adopted by another family shortly after being born.
Benjamin David was also a musician, who studied percussion at Juilliard, played for Broadway shows and was in the US Army Band.