Napoli enjoyed fine foods, good wines, and a life of luxury. Jimmy was known as a "Gentleman's Gentleman". Intelligent and well-respected, Napoli maintained ties to most of the major crime families. Napoli was married to Jeanne Napoli, a former night club singer and theatrical producer. In 1984, Napoli invested $250,000 in an unsuccessful musical biography of actress Marilyn Monroe that starred Jeanie.
In 1969, Napoli was indicted for fixing several boxing matches. In the 1970s, Napoli reportedly operated the largest numbers operation in the United States. By 1976, the operation allegedly employed 2,000 people and grossed over $150 million a year. Napoli apparently ran this huge operation from a lounge in the Greenpoint section of Brooklyn. In 1978, Napoli was convicted for gambling and sentenced to five years in federal prison.