After the Jay Treaty in 1795, the Baby family left the Detroit area and moved to Sandwich (now Windsor). (Today his house is owned by the Ontario Heritage Trust and currently houses government offices.) Over the years, the family acquired large amounts of land in the western region of Upper Canada. Baby was put in charge of the 1st Kent militia. During the War of 1812, Sandwich was seized by the Americans and Baby was later taken prisoner at the Battle of the Thames. During the American occupation, his property suffered extensive damage.
In 1815, he was appointed Inspector General and moved to York (now Toronto), where he was part of the ruling clique known as the Family Compact. In 1823 he represented Upper Canada in resolving a dispute with Lower Canada over the sharing of customs revenues. A Roman Catholic, he helped establish the first Catholic church at York, St. Paul's.