The Colored School was built at 702 E. Convent Street, in the Diamond Hill area of Victoria, which had been settled by freedmen after the Civil War.

[3] Twenty-five years after the legalization of slavery, at the outbreak of the Civil War, slaves equaled more than half of Victoria County's population.

[5] The first principal of the Victoria Colored School was Frederick William Gross (1861–1915), a native of Marshall, Texas, with degrees from several well-known universities.

[6] He encouraged literary pursuits and stressed academic performance among the students and, under his leadership the first six years, the school became recognized among black colleges.

It was eventually revealed that a lack of qualified teachers, space, and science equipment had been cited as reasons for the school's failure to gain accreditation.