Constructed on reddish stone on a rocky promontory at the far north of the island, the castle was originally cut off from the rest of the island by a moat, traces of which still remain. Although most of the roof caved in long ago, much of the lower part of the very substantial outer walls is intact, with the remains lying on different levels of the site.
The building of the castle was ordered by Portuguese commander Afonso de Albuquerque in 1507, when his forces seized the island for a short time, being then named "Fort of Our Lady of Victory"(Forte de Nossa Senhora da Vitória). Engaging his men of all ranks in the building works, Albuquerque faced a revolt and had to retreat. He came back and reconquered the island in 1515, when he concluded and renamed the fort. The Persian Safavid king, Abbas I of Persia (1587–1629), wanted to end Portuguese rule in the south and eventually managed to convince the English East India Company to allow its ships to cooperate with his land forces to seize the island from the Portuguese in 1622.
During the Habsburg Dynasty (1581–1640), the Portuguese built a fortress in Dibba Al-Hisn and a wall around the city. In August 1648, the Arabs besieged Muscat, Oman, and on October 31, 1648 a treaty was signed between the two opponents. The terms were as follows: the Portuguese should build the fortress of Kuriyat, Dibba Al-Hisn and Matrah (Oman).