A scrum-half, Albouy went through the different levels of Castres – his home town – youth teams before signing his first professional contract in 1998. On 2 February 2002, aged 22, he made his debut for the French national team during the 33–12 victory against Italy. As such, he took part in France's Grand Slam that year, but this cap turned out to be his last. Nonetheless, he enjoyed a successful club career in Castres, with whom he won the European Shield in 2003. In 2006, he decided to switch clubs and joined French giants Stade Français. He struggled in his first season there and was used mostly as a substitute behind Agustín Pichot and Jérôme Fillol, and therefore did not play a major role in Stade Français's title. After two more lukewarm seasons in the French capital, he returned to Castres at the start of the 2009–10 season. Again it looked like Albouy was going to start games on the bench more often than not, the spot in the starting XV being seemingly reserved to fellow international Sébastien Tillous-Borde, but the latter seriously injured himself early in the season, and Albouy took his place and the captaincy.