The first castle in Neath was built in 1120, and was located west of the river near the Roman Fort of Nidum. When Richard de Granville founded Neath Abbey close by, he abandoned the castle, and it may have been used by the monks as a source of building material.
A second castle, in what is now the centre of the town, is first documented in 1183. It was destroyed in 1321 and rebuilt in 1377, and that is the time when the great gatehouse was built, which is the main surviving feature. The castle was in use until the 17th-century, and has been a recreational area for the town since the 18th-century.