The beginnings of Southwestern Community College can be traced back to April 1961 when a need for skilled textile workers brought representatives of the North Carolina state Trade and Industrial Education Department to Jackson County.

The NC Board of Education provided needed equipment and logistics, and began hiring instructors, and an advisory panel was set up from local residents to oversee the project.

By 1963, Jackson County had obtained a 17-acre (69,000 m2) site for locating the college, and by May 1964, 90 percent of the block work for the building was completed, with local masonry students providing much of the labor.

The students, predominantly local residents who had been taking introductory classes in the previously established trade schools, were ready for hands-on experience.

In 1970 Southwestern began the process to become accredited by the Southern Association of Colleges and Schools, and received that status in December 1971.