Bayliss grew up in the NSW country town of Goulburn, before moving to Sydney to play for the NSW state team. A middle order batsman, he played in 58 first class matches, scoring 3,060 runs at an average of 35.58, and 50 List A limited overs matches. He was NSW's player of the season for 1989–90, scoring 992 runs including two centuries.
After the end of his playing career, Bayliss became a development officer for the New South Wales Cricket Association and coached the NSW second XI for a number of years. He replaced Steve Rixon as NSW coach in 2004-05 and led the state to the Pura Cup final where they defeated Queensland by 1 wicket. They couldn't defend the trophy the following season but did win the ING Cup.