The site was originally occupied by a smaller fortification, which was expanded into a castle between 1247 and 1325 by the Counts of Ortenburg, vassals of the Patriarchate of Aquileia. First mentioned in 1336 as castrum Grafenwarth, its current name is first recorded in 1449 as Costel, a corruption of the Latincastrum 'fortress'. After the extinction of the Counts Ortenburg on 28 April 1418, the Counts of Celje inherited their area holdings, expanding the castle into a formidable fortress and renaming it Schloss Grauenwarth, although the surrounding settlement retained the SlavicisedLatin name Kostel.
The castle and settlement were both surrounded by a high, two meter thick wall featuring five defence towers, built by order of Frederik II of Celje. The castle's purpose was the defence of the house's landholdings in Carniola; it also housed a local judiciary, and had its own dedicated execution site about 1 km away.
After the death of prince Ulrich II of Celje in 1456 and the extinction of the house, the castle was taken over by the Habsburgs, who eventually granted the settlement market rights, giving rise to a new colloquial name, Trg 'market town'.
During the 15th and 16th centuries, the castle was an important strategic fortification against Ottoman invasions. With many of the countries of southeastern Europe occupied by or paying tribute to the Ottoman Empire, Slovenia became exposed to further Ottoman inroads into Europe. The castle, standing along one of the Ottomans' common incursion routes into Slovenia, came under attack several times. Only in 1578 did the castle fall, when the garrison accepted supposed refugees from the Ottomans, but who opened the door that night to the Ottoman forces, who killed and captured the inhabitants of the castle and its village and the surrounding region. The depopulated area was then settled by numerous Uskoks.