Born in Belfast, Passmore worked as a salesman. In 1973, he became Grand Master of the City of Belfast Grand Orange Lodge. He also became Chairman of the Woodvale Unionist Association and a Justice of the Peace. In 1976, Irish Republican Army (IRA) members killed his father in an attack which Passmore claimed was aimed at him. In 1977, he gave a speech attacking the United Unionist Action Council and its plans for a general strike, claiming that it had been in discussions with the IRA.
Passmore was the Ulster Unionist Party candidate for Belfast West at the 1979 UK general election, taking second place and almost a quarter of the votes cast. He was then elected to the Northern Ireland Assembly, 1982, representing Belfast West. He stood again for the Westminster seat at the 1983 UK general election, but his share of the vote fell back to only 5.5%.