Schermerhorn was born in Schenectady, New York, where he studied clarinet, violin, and trumpet in school. At age 14, he forged a baptismal certificate to appear older so he could play in a dance band that played in night clubs. He soon created his own five piece band called The Blue Moods. He sang the lead and played trumpet for the group.
At 17, he was accepted into the New England Conservatory of Music, from which he graduated in 1950 with honors. He went on to play trumpet with the Boston Symphony Orchestra and the Kansas City Philharmonic among several other orchestras.
Schermerhorn was drafted into the U.S. Army and in 1953, while serving in Germany, he was assigned to be the conductor of the U.S. Seventh Army Symphony Orchestra. This was his first conducting position and he proved to be quite successful, winning the Elizabeth Sprague Coolidge Medal and the Harriet Cohen International Music Award for young conductors.
After leaving the Army, Schermerhorn studied and played under Leonard Bernstein at Tanglewood. At Tanglewood Schermerhorn won the Serge Koussevitzky Memorial Conducting Award for two consecutive years. This was a very influential time in Schermerhorn's life. He was quoted as saying of Bernstein, "He was my first real and certainly my most important teacher." Later in life, Schermerhorn would serve again under Bernstein as the assistant conductor of the New York Philharmonic.
In 1957, Schermerhorn was appointed to the position of music director of the American Ballet Theatre. He served in this position until 1968, and then again from 1982–1984. Schermerhorn, however, did conduct the 1977 television production of The Nutcracker, starring Mikhail Baryshnikov, Gelsey Kirkland, and the American Ballet Theatre. He also conducted other ballets that Baryshnikov appeared in during the 1970s, such as Twyla Tharp's Push Comes To Shove.
Kenneth Schermerhorn was music director of the New Jersey Symphony Orchestra from 1963 to 1965.
In 1968, Schermerhorn became the music director and conductor of the Milwaukee Symphony Orchestra. During his time there he was awarded the Sibelius Medal in 1979 from the Finnish government for his outstanding performance of works by Jean Sibelius.
In 1983 Schermerhorn joined the Nashville Symphony Orchestra as music director and conductor. He is widely credited with raising the level of excellence of the arts in Nashville. The Schermerhorn Symphony Center in Nashville, Tennessee, is named in his honor. The name of the Symphony Center was announced before his death.
Schermerhorn was also the music director of the Hong Kong Philharmonic Orchestra between 1984 and 1988. He helped improve the orchestra's quality and made several recordings with the orchestra; most notably, he conducted the orchestra on its debut 1986 tour of the People's Republic of China, which featured soloists Stephanie Chase and Li Jian and garnered worldwide media attention.