The town had a large Jewish population prior to World War II. In 1900 (when part of the Russian Empire) the town's Jewish population numbered 1,174. The Yiddish name for the city was סרעדניק (Srednik), corresponding to the Polish and Russian names Średniki and Средники (Sredniki). The name Seredžius is believed to be derived from середа sereda, a word meaning "Wednesday" in many Slavic languages. This is probably because of the markets held there on Wednesday.
View of Seredžius with Palemon Hill in the background
Southeast of the town, archaeologists discovered graves from the 3rd–4th centuries. The town also has a large hillfort, identified by historian Zenonas Ivinskis as Pieštvė, which was attacked by the Teutonic Knights numerous times in the late 13th and early 14th centuries. According to the Palemonids legend, noble refugees from the Roman Empire settled on the hill, now named after Palemon, and established the Grand Duchy of Lithuania. After the Battle of Grunwald, the location lost its military purpose and became private property of the Sapieha family. The Sapiehas built a residential palace, which did not survive. The town's first Catholic church was built around 1608–1612. The church was destroyed in 1829 after a landslide caused by extensive flooding. The residents built a wooden church, which was replaced by a Neo-Renaissance church dedicated to John the Baptist in 1913.
Seredžius was the birthplace of the American singer, comedian, and actor Al Jolson, born into the town's Jewish community in 1886 as Asa Yoelson.