Ivey became a noted Olympic and college swimming coach after his own competition swimming career ended. From 1974 to 1981, he was the head coach of the Santa Clara Swim Club, succeeding George Haines. Three of his Santa Clara swimmers qualified for the 1976 Summer Olympics. In 1981, he became the head coach of the Concord Pleasant Hill Swim Club. From late 1988 to mid-1990, he coached the elite Etobicoke Swim Club in Toronto, Ontario. He served as an assistant coach for the U.S. Olympic Team at the 1988 Summer Olympics. Ivey was chosen to replace Randy Reese as the head coach of the Florida Gators swimming and diving team of the University of Florida, and led the Gators women's team from 1990 to 1993. During his three seasons as Florida's coach, the Lady Gators swimmers won the Southeastern Conference (SEC) championship three consecutive years, and finished third, third and second nationally at the NCAA Women's Swimming and Diving Championships. He was also recognized as the SEC Coach of the Year for three consecutive seasons.
Despite his stellar three-year record of SEC championships and top-three NCAA national performances, the University of Florida Athletic Association released him in October 1993 following an episode of the ESPN television show Outside the Lines which recounted Ivey's history of romantic involvement with several of his previous swimmers before he became a coach at the University of Florida, and made allegations of sexual harassment against him. Ivey had been previously married three times, including his second wife who was an 18-year-old swimmer at the time he married her. Ivey denied the charges of misconduct, saying "I was told that putting my arm around a girl and using foul language was deemed reason enough" for the University of Florida to fire him. ESPN did not interview Ivey, nor did he answer on air any of the allegations by ESPN. His Florida women's swimmers issued a unanimous statement supporting him, and stated publicly they did not complain about nor witness any inappropriate behavior.
Ivey later coached the Trinity Prep Saints swimming and diving team of Trinity Preparatory School in Winter Park, Florida, and its affiliated club team, Trinity Prep Aquatics, during the late 1990s. Most recently, from 2003 to 2006, Ivey coached swimming at the Episcopal School of Jacksonville, Florida.
On December 24, 2013, USA Swimming, the national governing body for competition swimming in the United States, officially banned Ivey for life based on evidence that he had improper sexual relations with one or more swimmers while he was their coach.