Gangō-ji initially held as many as seven halls and pagodas in its precincts, which occupied a wide area within what is now Naramachi, the preserved district of modern Nara city. The original architecture was however, lost over time and notably in fires during the 15th through 19th centuries (Muromachi and Edo periods respectively).
Most of the destroyed complex ground has been altered and melded with parts of Naramachi over the course of time. Another small part of the temple remains today as the other Gangō-ji, of a Kegon school but with few remainders in terms of architecture.
The Man'yōshū includes a poem attributed to a monk of Gango-ji. This poet laments that, having attained enlightenment, his greater understanding remains unnoticed by others in the streets of Nara. His poem may perhaps bemoan his undervalued condition—and yet, in a modest way, his words transport contemporary readers momentarily back to share his quiet, 8th century perspective: