The inspiration for the breed was Pogo, a chestnutstallion foaled in 1960, a small crossbred horse, thought to have been sired by a Welsh pony, out of an Arabian/Tennessee Walkermare. He had a natural singlefoot gait. His owner, the late William M. Pugh, intrigued by the good-natured, naturally gaited stallion, developed a breeding program for small gaited horses, by crossing Pogo and his descendants with other Arabians, Tennessee Walkers, Morgans, and American Saddlebreds as well as Hackney ponies, and Welsh ponies. Horses that met Pugh's criteria for conformation, disposition and gaited ability were bred on. The registry began with two foundation stallions, Shadow of the Ridge and Pugh's Red Cloud.
The Virginia Highlander Horse Association was formed in the early 1990s to promote the Virginia Highlander breed through registry and education. Twenty-two horses (20 mares and 2 stallions) made up the original foundation stock. By June 2004, there were more than 130 registered Virginia Highlanders above and beyond the original foundation animals.