Taylor was born in Springfield, Illinois, as the fourth child and son of the seven children of Elizabeth Mary (née Rosemond) and Francis Marion Taylor.

His brothers were David, John A., Francis M., Willie, and Edward Taylor.

Francis began dealing in art in New York City for a wealthy uncle by marriage, Stephen Howard Young,[2][3] later, a friend of Dwight Eisenhower after World War II.

In April 1939, five months before the outbreak of World War II in Europe, they returned to the United States.

Taylor later ran an art gallery at The Beverly Hills Hotel in California.