Ya'alon served as the commander of Sayeret Matkal and lead it to many notable achievements, for which the unit received four recommendations of honor. In 1990 Ya'alon was appointed as the commander of Judea and Samaria division (אוגדת איו"ש). On December 101992 Ya'alon killed a Palestinian Islamic Jihad terrorist using hand grenade after the terrorist shot and killed YAMAM operator who tried to arrest him.
He was appointed Chief of Staff of the Israel Defense Forces (IDF) on 9 July 2002, and served in that position until 1 June 2005. The major focus throughout his tenure as Chief of Staff was the army's effort to quell the Second Intifada.
In February 2005, Defense Minister Shaul Mofaz decided not to prolong Ya'alon's service as Chief of Staff for another year. This marked the climax of tensions between Mofaz and Ya'alon, which had arisen partly through Ya'alon's objection to the Gaza disengagement plan. On 1 June 2005, Ya'alon ended his military service and Dan Halutz, his successor, oversaw the disengagement.
As Defense Minister Ya'alon decided to continue to manufacture and purchase Merkava tanks for the IDF, after the whole project was in question due to budget issues and the overall discussion of the necessity of tanks in modern battlefields. During Ya'alon's tenure foreign sources claimed that the Israel Air Force attacked several time advanced weapons deposits in Syria before they were transferred to Hezbollah.
On 27 August 2002, he told the Haaretz newspaper: "The Palestinian threat harbours cancer-like attributes that have to be severed. There are all kinds of solutions to cancer. Some say it's necessary to amputate organs but at the moment I am applying chemotherapy." In January 2004, he publicly stated that the thirteen Sayeret Matkal soldiers who refused to serve in the Israeli-occupied territories were taking the unit's name in vain.
In January 2008, during a discussion at the Interdisciplinary Center, Ya'alon said "There is no way to stabilize the situation all over the world and especially in the Middle East without confronting Iran." According to The Sydney Morning Herald Ya'alon said: "We have to confront the Iranian revolution immediately. There is no way to stabilize the Middle East today without defeating the Iranian regime. The Iranian nuclear program must be stopped."
When asked whether "all options" included a military deposition of Ahmadinejad and the rest of Iran's current leadership, Ya'alon told The Herald: "We have to consider killing him. All options must be considered."
In August 2009, Ya'alon visited the ruins of Homesh that was evacuated in Israel's unilateral disengagement plan in 2005 and toured Israeli settlements in the north of the West Bank, considered as un-authorized outposts. He said that these communities are all legitimate and should not be called "illegal." In addition, he participated in a convention of Manhigut Yehudit ("Jewish Leadership"), the more right-wing Settlers' segment within the Likud right-wing Party, in which he condemned the disengagement plan, called Peace Now a "virus" and said that "We become accustomed to Arabs being permitted to live everywhere, in the Negev, Galilee, Nablus, Jenin, and [on the other hand] there are areas where Jews are not allowed to live. We caused this." He also stated that, "regarding the issue of the settlements, in my opinion Jews can and should live everywhere in the Land of Israel. Now, ... first of all, every settlement needs to get the approval of the authorities, and what goes up on the spot, in contradiction to these decisions and so on is not legitimate. It's against the law".
Later, after meeting with PM Netanyahu, Ya'alon retracted parts of his statements and said that he "recognized the importance of democratic discourse and respecting other opinions." Ya'alon explained that, indeed, all Israelis want peace, now. He stressed, however, the need to accept the fact that peace won't come immediately, otherwise it "hurts Israel." Ya'alon stated that, in his view, the way of thinking that Israel just needs to give one more piece of land and then it will have peace is a kind of "virus."