Clifford played guitar as a child and won several talent competitions as a teenager. He signed to Bow Records at age 15, releasing a few singles but finding no success. After signing with Columbia Records, he released the single "Hello Mr. Moonlight", which did not chart. The follow-up, "Baby Sittin' Boogie"/"Driftwood" (though "Driftwood" was technically the B-side the record tends to be regarded as a double-A-side), became a crossover hit in the U.S. in 1961, peaking at #6 on the Billboard Hot 100, #27 R&B, and #28 Country. The record went on to sell over one million copies, and as a result of its success, Columbia tried to groom Clifford as a heartthrob pop singer. He appeared on TV with Perry Como and Merv Griffin and on American Bandstand, and toured the United Kingdom with Freddy Cannon and Dion. Clifford's fame was short-lived, however; further singles went nowhere (a remake of Kay Kyser's "Three Little Fishies" hit #102) and Clifford soon found himself without a recording contract.
After serving in the National Guard, Clifford moved to Los Angeles, California, and found work as a songwriter, writing tunes sung by Keith Barbour, Petula Clark, Clyde McPhatter, Lou Rawls, Leon Russell, Freddie King, and Kris Kristofferson. Later in the 1960s, he was involved with a band called Carp (including actor Gary Busey and songwriter Daniel Moore), who released one album on Epic Records in 1969. After a stint as one third of the group Hamilton Streetcar (with John Boylan of "Appletree Theatre" fame, and original group member Ralph Plummer—the group was originally named on behalf of former member John Burge, aka Ian Hamilton) which in 1969 recorded a self-titled album for Dot Records (Dot DLP25939), he also did recording sessions in Tulsa, Oklahoma, with former Beach Boy David Marks, but these were never issued.
Marks, Clifford, and Moore reunited in the 1990s and released an album called Work Tapes; Marks and Clifford continued to tour together along with Clifford's two sons until 1997. After recording in Denmark, another LP, Norse Horse, was issued.
In 2003, the 28-track CD More Than Just Babysittin': Complete Recordings 1958-1967 was released. The songs are digitally remastered, and some striking differences are apparent, including a fade-out ending on "Driftwood".
In 2009, the rap duo Method Man & Redman released the track "Four Minutes to Lock Down" (feat. Raekwon & Ghostface Killah), which contained a sample of the song "Echo Park," written by Clifford, as recorded by Johnny Maestro & The Brooklyn Bridge.
In 2011, Clifford reformed the band he had started with his two sons, Reese and John, along with Guido Bryant on guitar, Jon Greathouse on keyboards, and Zach Mathews on harmonica, and recorded and released a new CD, Bright Lights Shine, now available at CD Baby, iTunes and amazon.com.