A Troupe system is a way of playing role-playing games which spreads the game master's responsibilities among each of the players. The term was coined in Ars Magica. It is also known as collaborative role-playing, a term used by other games with a similar mechanism.
In a "traditional" role-playing game, one person typically acts as the gamemaster (known as Story Guide or SG in Ars Magica), and largely controls what happens in the game-world, what non-player characters do, and how the world at large reacts to the actions of player characters. However, in "Troupe" style play there is a presumption that different players will be SG at different times, when the game strays into their area of responsibility. There are multiple SG's who apportion various parts of the game-world and have more-or-less full control within their domain. One player may be the SG for a particular noble (the Duke of Burgundy, for example) and any adventures (stories) played in the Duchy of Burgundy or where the Duke is the antagonist. Another may control all the faeries in the campaign, and most or all faerie-focused stories. Most often, one player will be the "alpha" SG and coordinate overall interactions and continuity between these "beta" SG's and their various spheres of play.
In addition, players may create a set of characters for use in different contexts. A second assumption carried across most of the editions of the game is that the 'primary' characters who study and utilize the magical arts will gather together in relatively small groups called "Covenants", which may also refer to the structures in which said characters reside and research. Such an environment, very similar to an isolated outpost of mundane humanity, requires a support staff to fulfill the various mundane responsibilities that the mages pass over in favor of their magical studies: cooking, cleaning, farming, physical security and so on. The individual members of this staff are referred to as 'grogs' and collectively as a 'turb of grogs', and constitute a second context of characters which the players may generate and utilize within the game. Ultimately, this allows the players to incorporate mundane mortals engaged in historically-accurate mundane occupations into the 'high fantasy' setting of the game without sacrificing either the fantastic or historical elements.
The "Millington model"
In one model, attributed to Ian Millington, he identifies four roles normally performed by a GM: Chairperson (responsible for the order and focus of the players), Referee (responsible for arbitrating the game), Game Engine (responsible for interpreting what's happening in the game based on player requests and dice rolls), and Director (responsible for the story and the setting of the game). In a collaborative role-playing game each of these roles may be assigned to a different player, or may be shared among several players. There is therefore a continuum between the (mostly theoretical) Millington model of spreading responsibility out to multiple people at the same time and the Ars Magica model of having one SG at a time.
The troupe system described above should be distinguished from the more commonly used term "troupe-style play" which was also popularized by Ars Magica. Troupe-style is generally use to mean that, instead of, say, a party of adventurers, with each player playing a single character, the players together form a larger group. In Ars Magica, for instance, each player creates a Wizard character. They then all create a lieutenant for that wizard, and then, finally, a number of grunt-style retainers.
Both terms (Troupe system and Troupe-style play) are often used interchangeably and neither has a clearly established definition. It would be fair to say that when term troupe-style or troupe system is used in the role-playing context the most common meaning is the troupe-style play with players playing multiple characters, rather than the system of rotation gamemasters.