In place of the modern town, in the 15th century there was the Mažieji Pienionys (Little Pienionys) estate. At the end of 15th century Grand Duke of Lithuania Alexander Jagiellon transferred the estate to Stanislovas Kovarskis, the treasurer of the king. As he had no successors, the estate was inherited by his brother, Andrius Kovarskis, the canon of the Vilnius Cathedral. Thus the name of the Little Pienionys estate was changed to Kavarskas.
In written sources Kavarskas was first mentioned in 1538. In that year a church was built. In the 16th century, Kavarskas and the surrounding area belonged to nobles Astikai, later – Ogiński (Oginskiai), Tyszkiewicz (Tiškevičiai), and Siesickiai. In 1956 Kavarskas was granted town rights.