[4][5] After graduating from Portsmouth's I. C. Norcom High School in 1944, Holley served in the United States Army during World War II, stationed in Camp Livingston in Louisiana.

[6] During a reception in the late 1950s, Holley met Virginia Union University student Mary Walker; the couple would marry in 1960.

[6] Holley was active during the Civil Rights Movement in the 1950s and 1960s,[9] and played an integral role in the desegregation of Portsmouth, winning court battles which allowed for the equal use of the city's libraries, hospitals, restaurants and golf courses.

[10] During the course of his involvement in the civil rights movement, Holley entertained Martin Luther King Jr. at his home on multiple occasions.

[16] His successor, Gloria O. Webb, former chairperson of the School Board, not only became Portsmouth's first female mayor, but also served uncontested for a decade.

[3] This second recall effort began in 2009 after an assistant made a confidential complaint of verbal abuse and of being required by Holley to perform his personal errands while working on city time.

The accusations (leaked to the press) prompted the Portsmouth City Council to fine Holley $2,500 and ask him to retire, citing a pattern of mistreating subordinates.