The town is built along the narrow valley of the Cherna ('black') and the Byala ('white') river in the central Rhodope Mountains.

For a while during the 14th century it came under the control of the Bulgarian feudal lord Momchil, alongside the whole Rhodope mountains, before eventually being subjugated by the Ottoman Empire.

The area was liberated by the 21st Sredna Gora Regiment led by Vladimir Serafimov in 1912, during the First Balkan War.

The modern town of Smolyan was formed by the merger of three existing villages — Ustovo, Raykovo and Ezerovo — in 1960.

The largest church in southern Bulgaria, the Cathedral of Saint Vissarion of Smolyan, was inaugurated in the city in July 2006.