The area of the present village was mostly unpopulated and resorted under the rule of the city of Osor on northern Lošinj which counted 20,000 inhabitants at that time.

For a period Veli Lošinj was the seat of the bishopric due to the plague visiting nearby Osor.

In the 15th century, the still-standing defensive tower was built, erected to defend the harbour and the settlement, but also to protect the locals from pirate attacks.

In the 17th century, the town continued to develop in a circular form around the bay of Veli Lošinj.

It became urban in appearance and Baroque in style with a central square near the sea and two main streets of which one leads to Velo Selo, and the other to the settlement of Rovenska, where a fishing village with stone houses was erected.

Antun Opat Pustinjak (Saint Anthony of Egypt) was built in place of an earlier smaller church in the 15th century.