At the end of the 12th century A.D., the Lehola hill fort of ancient Estonians was built at a location (now known as Lõhavere) approximately two kilometres north-east of the present town border.

Not much except the hill remains of it but the place (known in Estonian also as Lembitu linnamägi) is still visited as a tourist attraction.

The town began to develop around the (now Lutheran) Church of Great Saint John the Evangelist (Groß Sankt Johannis in German), built before the year 1300.

The present name, "Suure-Jaani" (literally meaning "Great John's"), which came into widespread use for the locality only in the 1880s, when it developed into a village in its own right, is an Estonian derivative from the German name of the church.

Suure-Jaani is situated in Sakala Upland, offering picturesque views with its park, two artificial lakes and church in the centre of the town.

During the Estonian national awakening period in the 19th century, Suure-Jaani parish was one of the regional centres of educational, cultural and social life (largely as a result of the efforts of schoolmaster and musician Joosep Kapp and also e.g. as the parish of origin of the first Estonian professional painter Johann Köler, who both actively contributed to the development and promotion of Estonian culture).

Suure-Jaani is also the place of establishment of the International Artur Kapp Society, a non-profit foundation for promoting the works and life of composer Artur Kapp and other composers related to Suure-Jaani, such as Mart Saar.