Abley spent some of the 1949 VFL season playing reserves football with Hawthorn Football Club and turned his back on a possible career in the seniors when he moved to Adelaide, South Australia. Abley could easily have played for Glenelg (as his younger brother Kevin did) instead of Port as after he arrived in Adelaide the area he planned on residing in was part of Glenelg's recruiting zone. Port Adelaide officials, at the suggestion of Bob McLean, hastily arranged alternate accommodation in the hope of acquiring his services and Abley debuted for Port Adelaide in 1950.
A fullback, Abley first played in that position for Port in a "Challenge match" in Broken Hill at the end of his debut season and after impressing was kept there for the remainder of his career. He was a member of the famed Port team which won six premierships in succession from 1954 to 1959, an Australian record.
Abley was a regular South Australian interstate representative and played a total of 23 games for his state. On three occasions Abley was selected as an All-Australian for his performances at the carnivals, in 1956, 1958 and 1961.
He has the habit of worry that makes for perfect concentration,
sure-footedness (never a reckless throw for the ball),
and the safest, most relaxed mark in the business.
In 1959, Abley was named as a life member of the Port Adelaide Football Club because of his talent, his warmth off the field and being one of only six players to play in all of Port Adelaide's premierships in the 1950s (1951, 1954, 1955, 1956, 1957, 1958 and 1959).