First built during the Huangyou era (1049–1053) of Song Dynasty Emperor Renzong, the village was originally called Xichuan (West River), owing to the various water courses flowing through it.
The rise of the village was closely tied to the fortunes of the Hu family. By 1465 CE, during the Ming Dynasty (1368–1644), family members had started in business as merchants, leading to construction of major private buildings and a public infrastructure. By the middle of the 17th century, the influence wielded by members of the Hu family expanded from commerce into politics. The prosperity of Xidi peaked in the 18th and 19th centuries, at which time the village comprised about 600 residences.
The street pattern of Xidi is dominated by a main road which runs in an east-west direction and is flanked by two parallel streets. These major streets are joint by many narrow alleyways. Small open spaces are confined to areas immediately in front of the main public buildings, such as the "Hall of Respect", the "Hall of Reminiscence", and the "Memorial Archway of the Governor".
Today, 124 well preserved wooden residences from the Ming and Qing dynasties with beautiful carvings form the major attractions. Many of these residences are open to the public.