Born in Ashkelon, Cohen served on the city's council and was its former deputy mayor.
He served as Secretary-General of Shas' educational institution (El HaMa'ayan). He was first elected to Knesset in 1996. In his time in Knesset he has served on the Finance Committee, Labor and Welfare Committee, Committee on Foreign Workers, Foreign Affairs & Defense Committee, Joint Committee for the Defense Budget, Special Committee for Discussion of the Security Service Law Substitute Member, Constitution, Law and Justice Committee, and the House Committee. He has also been a member of the Knesset Forum on the Middle East lobby, and is currently a member of the Lobby for Ethiopian Immigrants.
He became the Minister of Religious Affairs in July 1999, and resigned a year later. He served as the Deputy Minister of Finance from 2001–2003. In 2006 he was appointed the Minister without Portfolio, and is in charge of religious councils, a role converted to Religious Services Minister in January 2008.
In 2007 he criticized the US report that Iran had stopped its nuclear program in 2003. He said the report was "ordered by someone who wants dialogue with Tehran". He compared the event to when the US got intelligence in World War Two that said there were trains going to concentration camps, and they said they were being used for industrial purposes. In 2009, in an interview with the Spiegel, he threatened to suspend relations with the Vatican, following the lifting of the excommunication of controversial bishop Richard Williamson.
He retained his seat in the 2009 elections, having been placed third on the Shas list. He lost his cabinet portfolio, instead becoming Deputy Finance Minister.