In the 1870s Mansel tinplate works was built in the area, and its proprietors, Col. D. R. David and Sir Sidney Byass encouraged the local workers to form a rugby team.

[5] In 1907 Aberavon RFC moved to the Central Athletic Ground and in 1913 Miss Emily Charlotte Talbot gave exclusive rights to the club to use the pitch and granted them a lease for 39 years, giving needed stability.

In 1952 Aberavon RFC changed their shirt colours for the final time to red and black hoops; and later that year succeeded in becoming the absolute owners of the Talbot Athletic ground.

The expansion of the local steel works in the 1950s also aided in bringing new enthusiasts to the area and this was reflected in one of Aberavon RFC's most successful periods.

As a result, Aberavon struggled for much of the following season, although they did upset the odds by defeating a touring Western Samoan team by 22 points to 11 on 26 October 1988.

When rugby union turned professional in the mid-1990s, it quickly became clear that Aberavon RFC had neither the financial resources nor the ambition, at least at committee level, to seek a return to the upper echelon of the Welsh club game, instead appearing to settle for a regular position in the second tier, much to the frustration of their loyal supporter base.

With a management structure more aligned to the professional era now in place, there followed a highly successful period under coach Chris O'Callaghan (who had been one of those players to have left the club in frustration twelve years earlier, and who had been approached by the club's committee in November 1999 to take the helm following a string of disastrous results), but promotion to the game's top tier was then repeatedly blocked by successive rule changes made by the Welsh Rugby Union (on one occasion a matter of a few weeks before the season ended), until the game in Wales was restructured in 2003, at which point Aberavon RFC found itself frustratingly outside looking in as the new professional "regions" were set up.

Due to the clubs' original January fixture being rearranged, the league title would be decided in a winner-takes-all clash at the Talbot Athletic Ground on 13 May 2003.

Since then the club has gradually established a good working relationship with the Ospreys regional team, with O'Callaghan eventually stepping down from the coaching role in 2004 to be succeeded by Kevin Hopkins, who in turn was succeeded by current incumbent Simon King, and currently working with the region in the development of talented young players.

The more generally accepted view, however, is that the nickname was coined by South Wales Evening Post reporter Bill Taylor during the 1920s, when he dubbed the highly successful Aberavon team of that era "The Wizards of the West".