He met his first wife, Carrie Marcus Neiman, in Dallas, Texas while they were working at A. Harris & Company.
He traveled widely as well to ensure Neiman Marcus could bring the best clothing money could buy to Dallas from New York and elsewhere.
They divorced and Herbert bought out Al's share for $250,000 on the condition that he couldn't legally compete with Neiman Marcus in Dallas.
He married Dorothy Squire, a fashion model, in 1938 and they adopted World War II refugees Diana and Ursula Woolf after discovering they were unable to have their own children.
His former nephew, Stanley Marcus, visited him at a Masonic home in Arlington, Texas, in October 1970, fearing few others would before his death.