The first to build a palace with a church on the Castle Hill of Esztergom, well after the Romans, was Grand Prince Geza. His son Vajk was born in the building on the northern side of the hill, and was also baptised and crowned here. On the southern side of the hill, which was originally separated from the royal-clerical centre by a mound, the first houses were built around the middle of the 11th century, however these probably served as the residency of the Bailiff of Pilis, and were outside the royal centre.
The first reconstructions took place around the turn of the 11th and 12th centuries, then, at the end of the 13th century, during the rule of Béla III and his son Imre, the block of the present royal palace was erected in several phases, and, by the end of the century, the royal court had been moved here.
In 1256, after the construction of the Palace of Buda, the residency on the southern side of the hill was given to the Archbishop of Esztergom, who used it as his seat until 1543, when the church escaped from the Turks’ advance to Nagyszombat.
In the 16th century the surviving buildings were populated by the court of the Bey of Esztergom, even though these buildings were also damaged during the sieges of 1594 and, mainly, of 1595.
The ruins of the dwelling tower, the chapel and the chambers of the archbishop were buried with earth, to form the base of a cannon stand.