The site used to be the Ōmiya Palace (大宮御所,Ōmiya-gosho?), the residence of Empress Teimei, the consort of Emperor Taishō. After her death at the palace in 1951, the site of the palace was converted to the Crown Prince's residence.
Following tradition, the Crown Prince resided in the Akasaka Palace before moving to the more modern and smaller East Palace. Responsible for the upkeep and organisation of the palace is the Palace Chamberlain (東宮侍従, Tōgū-jijū). The Chief Chamberlain to the Crown Prince is the head of the Crown Prince's Household and reports to the Imperial Household Agency.