Johnson was born in Snow Hill, Maryland, on October 26, 1899, to Annie and William Johnson. His family moved to Wilmington, Delaware, when he was five years old. He attended Howard High School but quit to work on the docks in New Jersey. Johnson was 5 feet, 11 inches tall and weighed 150 pounds.
In 1919, Johnson played for a Philadelphia semi-pro team, the Madison Stars. He was acquired by a Negro major league club, the Hilldale Daisies, for $100, and played for Hilldale from 1921 to 1929. During that time, he was nicknamed "Judy" because he resembled Judy Gans, a player for the Chicago American Giants. Future Baseball Hall of Famer John Henry Lloyd became Johnson's mentor and taught him how to play third base.
When the Hilldale club folded, Johnson became the player-manager of the Homestead Grays. There, he discovered and became a mentor to future Hall of Famer Josh Gibson. Johnson then ended his career playing for the Pittsburgh Crawfords from 1932 to 1936. He was the team's captain and helped them win the pennant in 1935. Johnson had a career batting average of .298 in the Negro major leagues.