Kaye is the son of Arnold and Elaine Friedman. He is the oldest of their three sons. Kaye's younger brothers are Seth and Jesse. He grew up in Flushing, Queens, and then moved to Great Neck, Long Island, with his family when he was a teenager.
Kaye adopted his current surname in the 1990s. He says he registered with the union for television performers with the surname Kaye to differentiate himself from the 27 David Friedmans already in the union.