The town of Llandovery was decided upon "because of its central position and because of easy communication with all parts of South Wales".

Between 1901 and 1903, alterations were made to the existing school, and new buildings were added, by the Lancaster architects Austin and Paley.

The additional buildings included the east range, a school house, and a dining room, at a cost of about £10,000.

The college is equipped with over 45 acres (18 ha) of playing fields, an all-weather pitch, climbing wall, gym and weights room and a 9-hole golf course.

Although St David's College, Lampeter is accepted as the first establishment to play rugby, Llandovery was one of their first opponents, were present at the formation of the Welsh Rugby Union in Neath in 1881 and provided two players to the very first international game Wales played.