The tower was erected on the Turtle islet, the former fishing site under king Le Thanh Tong. Under the Restored Le Dynasty (17th and 18th centuries), the Trinh Lords had Ta Vong Temple built on the islet, which disappeared during the Nguyen Dynasty.
After the French conquered the Hanoi Citadel, most of the population around the lake fled. Vietnamese officials also fled their offices. There stayed only Nguyen Ngoc Kim, who was the mediator between the French troops and the Vietnamese.
- Trần Độ (Editor-in-chief). Văn hóa Việt Nam. Hà Nội: Ban Văn hóa Văn nghệ Trung ương, 1989. Page 101.