Gelgaudiškis is the Lithuanian name of the city. Versions of the name in other languages include Polish: Giełgudyszki, Russian: Гельгудишки Gel'gudishki, Belarusian: Гельгудiшкi Gel'hudishki, Yiddish: גלגודישק Gelgudishk.
Before the end of the 14th century the area of Gelgaudiškis settlement and manor, positioned on the upper bank of Neman, belonged to the Grand Dukes of the Grand Duchy of Lithuania. In 1504, King and Grand Duke Aleksander Jagiellon gave it away as a gift to his royal secretary Sapieżyc, progenitor to the szlachta family of Sapieha. Afterwards, the grange belonged to families of Massalski, Dembiński, Oziembłowski, Giełgud, Czartoryski, the Prussian barons von Keudell, and finally Komar family (till the beginning of World War I). In the interwar period, the establishment was nationalized and housed an orphanage and later a school.
The classicist palace was built in the first half of the 19th century by the family of von Keudell. After the fire of 1979 the palace was rebuilt and since remains unoccupied. It is surrounded by a garden and forested park with an area of 112 ha leading to Neman River.
The city and nearby castle is mentioned in John Gielgud: The Authorized Biography, (by Sheridan Morley), as being the actor's ancestral home and from where his family name originated.