The island was first used as a fishing site. In the 1400s, under Lê Thánh Tông a tower was erected to improve the comfort of the king's fishing. In the 17th and 18th centuries, under the Restored Lê dynasty, the Trịnh lords had Ta Vong Temple built on the islet. In the 18th century, under the Nguyễn lords the temple disappeared.
In 1886, while Vietnam was occupied by the French, a musician who was secretly working for the French, received permission from the government to build a tower in the middle of Hoan Kiem Lake in honor of Lê Lợi, one of the most famous figures of Vietnamese history and one of its greatest heroes. Legends surrounding his life involving the sword, Thuận Thiên and Hoan Kiem turtle are associated with Hoan Kiem Lake and the island.
The musician planned to secretly bury his father within the tower. Residents of the city discovered his plans and removed his father's body from the structure. The three-story tower was still completed and was originally named Ba Ba Kim Tower.
- Trần Độ (Editor-in-chief). Văn hóa Việt Nam. Hà Nội: Ban Văn hóa Văn nghệ Trung ương, 1989. Page 101.
- "The Story of Hanoi's Statue of Liberty". Saigoneer. 29 May 2016.