Born in Azusa, California, Kilmer played baseball, football, and basketball during his athletic career at Citrus Union High School in Azusa, scoring more than 1,500 points in the latter sport while earning All-American honors. His baseball exploits saw him win All-Conference accolades, while the poor fortunes of his football squad saw him relegated to Third Team All-Conference recognition.
Kilmer then competed in football for one year at Citrus Junior College, where he scored six touchdowns and threw for 15 more, with a broken foot limiting his playing time for the Citrus basketball team. He transferred to UCLA in 1958 and played three seasons, closing out with an outstanding senior season in 1960 that helped him finish fifth in voting for the Heisman Trophy. During that stellar campaign, he threw for over 1,000 yards, ran for over 800, scored eight touchdowns and served as the team's punter. Kilmer was also awarded the 1960 W.J. Voit Memorial Trophy as the outstanding football player on the Pacific Coast.
After the conclusion of a UCLA career that saw him finish among the top five in school history for passing and rushing yards, as well as total offense, Kilmer played in the 1961 College All-Star Game, where he earned Most Valuable Player honors.
Kilmer was the eleventh overall selection of the 1961 NFL draft, taken by the 49ers. He was also selected in the fifth round of the 1961 AFL Draft by the San Diego Chargers, but signed with San Francisco on December 30, 1960. During that rookie season, he saw action primarily at running back, rushing for 509 yards and ten touchdowns, with his top performance coming against the expansion Minnesota Vikings on October 15 when he rushed for four touchdowns.
The following year, Kilmer's season came to an early end following a December 5 auto accident in which he fell asleep in his 1957 Chevrolet convertible and drove off the Bayshore Freeway into the San Francisco Bay. Suffering a fractured leg, Kilmer's injury was bad enough to also force him to sit out the entire 1963 NFL season. He was able to return the following year, but his production was limited. After he saw no action in 1965 and was involved in a training camp contract dispute the next season, he was placed in the 1967 NFL Expansion Draft.
On February 10, 1967, Kilmer was selected in the expansion draft by the New Orleans Saints and, despite originally being thought of as the team's third-best signal caller, became the team's starting quarterback for much of the next four seasons, although he shared time with Edd Hargett during his final two seasons. His most prolific performance during his four-year stint with the team came on November 2, 1969, when he threw for 345 yards and six touchdowns in a 51-42 win over the St. Louis Cardinals.
Frustrated after four years of the Saints' futility and sensing New Orleans would draft Ole Miss star Archie Manning with the second overall pick of the 1971 NFL Draft, Kilmer asked to be traded and was granted his wish on January 23, 1971, when he was traded to the Washington Redskins for linebacker Tom Roussel and two draft selections.
A short, but friendly dispute arose about who was the actual holder of Tom Dempsey's record-setting 63 yard field goal kick. For many years, it was widely believed that Saints quarterback (at the time) Billy Kilmer was the holder, but it was actually Saints' safety Joe Scarpati (#21), who was the holder of the (then) record-setting field goal. The dispute was finally put to rest, as still pictures and a few videos of that moment finally surfaced, showing that Scarpati (mentioned by name by veteran CBS announcer Lindsey Nelson, and verified by his jersey number #21) was indeed the holder of Dempsey's kick. A few weeks after the record-setting kick, Kilmer was traded to the Washington Redskins, where he achieved greater fame.
That status changed on September 4 when Jurgensen suffered a severe shoulder injury against the Miami Dolphins. Kilmer then led the resurgent Redskins to a 5-0 start, but a midseason slump cost them the opportunity to win the National Football Conference Eastern Division. Kilmer briefly lost his starting job as a result of the slide, but regained it after Jurgensen again injured his shoulder. Fans in Washington tended to be loyal to one quarterback or the other, sporting buttons at games that read "I like Billy" or "I like Sonny." Kilmer did not throw the tight spiral that Jurgensen did, but his wobbly passes often got the job done.
One year later, Kilmer led the Redskins to an NFC-best 11-3 record, while also leading the NFL in touchdown passes (19) and passer rating (84.8). After the season, Washington advanced to their first Super Bowl, which they lost, 14-7, to the unbeaten Miami Dolphins (16-0). The loss didn't dampen Kilmer's individual recognition as he was named to the Pro Bowl and the All-NFC Team.
That success would endear Kilmer to Allen, who remained loyal to the signal caller, despite determined challenges from Jurgensen and former University of Notre Dame quarterback Joe Theismann. Kilmer remained with the Redskins until 1978, serving as a backup to Theismann during that final year. Kilmer's demotion could undoubtedly be due to the dismissal of Allen at the conclusion of the previous season.
During his time with the Redskins, Kilmer became one of the few remaining users of a single-bar face mask on the helmet, as multi-bar face masks became the norm in the NFL. Theismann also wore the single-bar throughout his career.
Kilmer finished his 16-year NFL career with 1,585 completions in 2,984 attempts for 20,495 yards and 154 touchdowns, with 146 interceptions. He also rushed for 1,509 yards and 21 touchdowns, caught 27 passes for 288 yards and one touchdown, and punted the ball 16 times for 598 yards.
It was during the 1970 season that rumors stirred that Kilmer missed curfew before a game against the Denver Broncos and that his new Redskins teammates had nicknamed him "Ole Whiskey."
Kilmer’s most memorable night on the town came early Monday morning December 6, 1971 after beating the Giants 23-7. He got arrested at the Toddle House, a coffee shop in Arlington. Apparently, Kilmer was attempting to pay a $4 tab with a $100 bill and an argument ensued involving the waitress. A policeman showed up, and Kilmer, who had been drinking anything but coffee earlier that evening, told the policeman, “If you think I’m wrong, put me in jail!” The policeman locked Kilmer up for the night, releasing him in time to lead the Redskins into the playoffs against San Francisco. Kilmer “won the hearts of fans when he told the waitress she could keep the $100 as a tip”.
This arrest gave rise to the quarterback's image as a hell-raiser. That reputation had been noted by Saints owner John Mecom, Jr., who mentioned Kilmer's many late-hour visits to New Orleans bars while playing for the team.
On December 11, 1976, Kilmer was arrested for drunk driving less than 2 days before a game against the Dallas Cowboys.
Following his retirement, Kilmer stayed on the fringes while working for a gambling service that made selections on NFL games, but has since stayed out of the spotlight. Kilmer served as coach of the Shreveport Steamers of the American Football Association (a summer professional league) in 1979 and commissioner of the same league in 1981, but left as a result of numerous problems under his tenure as well as a lack of payment. He occasionally makes appearances in Mobile, Alabama in support of the GMAC Bowl.