The Damasd castle is mentioned for the first time in 1361, as a hunting palace of the king.
It remained in royal possession until 1523. In the period of Hungaryas Turkish conquest the castle of Damasd played the role of a significant garrison in the defense system of the country.
During the siege of Esztergom of 1543-44 the castle of Damasd got also destroyed. In 1581 it was rebuilt by the Turks. Throughout the 17th century the castle was repeatedly fortified, but eventually in 1641 it was demolished and never rebuilt afterwards. The medieval church of Ipolydamasd was located within the castle walls, but it did not survive either.
Today the ruins of the castle can be found on the north-western edge of the village, on the hill above the Ipoly river.