Kinsey held jobs on trans-Atlantic ships while young, studying while at port with Bill West in New York City and with local musician Tommy Webster in Birmingham. He had a close association with Ronnie Ball early in his life; the two even had a double wedding together. Kinsey led his own ensemble at the Flamingo Club in London through the 1950s, and recorded on more than 80 sessions between 1950 and 1977, including with Tubby Hayes, Bill Le Sage, Ronnie Scott, Johnny Dankworth, Joe Harriott, Lena Horne, Ella Fitzgerald, Ben Webster, Clark Terry, Harry Edison, Buddy DeFranco, Billie Holiday, Oscar Peterson, and Sarah Vaughan. He performed at European jazz festivals both as a drummer and as a poet. He did some work as a session musician in the 1950s and 1960s, playing on records by Eddie Calvert, Cliff Richard, and Ronnie Aldrich.
Kinsey also branched into composition; a string quartet composition of his is used in the short film On the Bridge, and he wrote arrangements for big bands in addition to music for over 100 commercials. Later in his life he wrote music for a musical based on the life of George Eliot.