Evolutionary game theory

Evolutionary game theory (EGT) is the application of game theory to evolving populations of lifeforms in biology. EGT is useful in this context by defining a framework of contests, strategies, and analytics into which Darwinian competition can be modelled. EGT originated in 1973 with John Maynard Smith and George R. Price's formalisation of the way in which such contests can be analysed as "strategies" and the mathematical criteria that can be used to predict the resulting prevalence of such competing strategies.

Evolutionary game theory differs from classical game theory by focusing more on the dynamics of strategy change as influenced not solely by the quality of the various competing strategies, but by the effect of the frequency with which those various competing strategies are found in the population.

Evolutionary game theory has proven itself to be invaluable in helping to explain many complex and challenging aspects of biology. It has been particularly helpful in establishing the basis of altruistic behaviours within the context of Darwinian process. Despite its origin and original purpose, evolutionary game theory has become of increasing interest to economists, sociologists, anthropologists, and philosophers.

The problem

John Maynard Smith

"An ounce of algebra is worth a ton of verbal argument" J.B.S. Haldane (as quoted by John Maynard Smith)

The need for evolutionary game theory in biology started with a problem. The problem was how to explain ritualized animal behaviour in a conflict situation; "why are animals so 'gentlemanly or ladylike' in contests for resources?" This was a problem that leading ethologists Niko Tinbergen and Konrad Lorenz were trying to address. Tinbergen proposed that such behaviour exists for the benefit of the species. Maynard Smith couldn't see how this reasoning matched with Darwinian thought as he understood it, where selection occurs at an individual level and therefore defections to self-interest are rewarded while seeking the common good is not. Maynard Smith, a former engineer and highly competent mathematician, turned to game theory to answer the question. The suggestion that game theory might solve the problem was made by George Price even though some unsuccessfully applied efforts by Richard Lewontin at using game theory had failed. Maynard Smith’s way of applying these mathematics proved to be a far more realistic model, and came to be a major mathematical contribution in the field of theoretical biology.

Classical game theory

Main article: Game theory

Game theory was conceived by John von Neumann to mathematically determine optimal strategies in situations in which a competition between adversaries is taking place. A contest involves a number of players, all of whom have a choice of moves for the game. Games can be a single round or repetitive. The approach a player has in making his or her moves constitutes the player's "strategy". Rules govern the outcome for the set of moves taken by the players and outcomes produce payoffs for the various players; rules and resulting payoffs can be expressed as decision trees or in a payoff matrix.

Classical game theory essentially requires that all of the players make rational choices (that is, making their strategic choice on a wholly rationally determined evaluation of probable outcomes). As a consequence, it is fundamental in game theory that each player must consider the strategic analysis that the players' opponents are making in determining that his or her own strategic choice is appropriate.

Adapting game theory to evolutionary games

On reflection, Maynard Smith realised that an evolutionary version of Game Theory does not really require that players act rationally – only that they have a strategy. The results of the game will test how good that strategy is. That is what evolution does – it tests alternative strategies for the ability to survive and reproduce. In biology, strategies are genetically inherited traits that control an individual's action – and strategies are algorithmic – just like computer programs. The key point in the evolutionary game theory model is that the success of a strategy is not just determined by how good the strategy is in itself, it is a question of how good the strategy is in the presence of other alternative strategies, and of the frequency that other strategies are employed within a competing population. It is also a question of how good a strategy plays against itself, because in the biological world a successful strategy will eventually dominate a population and competing individuals in it end up facing identical strategies to their own.

Participants in the evolutionary game aim to become more fit than competitors – to produce as many replicas of oneself as one can, and the payoff is in units of fitness (relative worth in being able to reproduce). It is always a multi-player game with a very large population of competitors. Rules describe the contest as in classical Game Theory but for evolutionary games rules include the element of replicator dynamics, in other words the general rules say exactly how the fitter players will spawn more replicas of themselves into the population and how the less fit will be culled out of the player population (expressed in a Replicator Equation). The replicator dynamics in essence models the heredity mechanism, but for simplicity leaves out mutation. Similarly, Evolutionary Game Theory only uses asexual reproduction for the sake of simplicity. Games are run repetitively with no terminating conditions. The results that are studied include the dynamics of changes in the population, the success/survival of strategies, and any equilibrium states reached. Unlike in classical Game Theory, players do not choose their strategy or have the ability to change it, they are born with that strategy and their offspring will inherit that same identical strategy.

Models

The Evolutionary game model Evolutionary game theory transposes Darwinian mechanisms into a mathematical form by adopting a System Model of evolutionary processes with three main components - Population, Game, and Replicator Dynamics. The system process itself has four phases:

1) The model (as evolution itself) deals with a Population (Pn). The population will exhibit Variation among Competing individuals. In the model this competition is represented by the Game.

2) The Game tests the strategies of the individuals under the “rules of the game”. These rules produce different payoffs – in units of Fitness (the production rate of offspring). The contesting individuals meet in pairwise contests with others, normally in a highly mixed distribution of the population. The mix of strategies in the population affects the payoff results by altering the odds that any individual may meet up in contests with various strategies. The individuals leave the game pairwise contest with a resulting fitness determined by the contest outcome – generally represented in a Payoff Matrix.

3) Based on this resulting fitness each member of the population then undergoes replication or culling determined by the exact mathematics of the Replicator Dynamics Process. This overall process then produces a New Generation P(n+1). Each surviving individual now has a new fitness level determined by the game result.

4) The new generation then takes the place of the previous one and the cycle begins again (and never stops). Mathematically speaking it is an Iterative process. Over time the population mix in such a system may converge to a stationary state – and if such a state cannot be invaded by any new “mutant strategies” it is by definition an Evolutionary Stable State (ESS)

It is important to realize that EGT is not just a specialist mathematical treatment of animal contests to determine their dynamics and results, but in a manner similar to the field of Evolutionary Algorithms, it is a comprehensive model that encompasses the Darwinian process itself – including the central tenets of competition (the game) natural selection (replicator dynamics) and heredity - all within the overall model. Therefore, it is a major vehicle to help understand and explain some of the most fundamental questions in biology including the issue of group selection, sexual selection, altruism, parental care, co evolution, and ecological dynamics. Much of the progress in developing understanding in these diverse areas has been aided by evolutionary game theory modelling and many of the counter intuitive situations in these areas have been explained and put on a firm mathematical footing by the use of these models.

The common methodology to study the evolutionary dynamics in games is through replicator equations. These replicator equations in the context of evolutionary biology show the growth rate of the proportion of organisms using a certain strategy and that rate is equal to the difference between the average payoff of that strategy and the average payoff of the population as a whole. Continuous replicator equations assume infinite populations, continuous time, complete mixing and that strategies breed true. The attractors (stable fixed points) of the equations are equivalent with evolutionarily stable states. A strategy which can survive all "mutant" strategies is considered evolutionary stable. In the context of animal behavior, this usually means such strategies are programmed and heavily influenced by genetics, thus making any player or organism's strategy determined by these biological factors.

Games

In Game Theory various games are mathematical objects of different character — in rules, relative payoffs, and attendant mathematical behaviours. Each "game" represents different situations — in the kinds of problems that organisms have to deal with, and the possible strategies that they might adopt if they are to successfully survive and reproduce. To achieve a better feel for the challenges of these different situations, evolutionary games are often given rather colourful names and “cover stories” which quite effectively describe the general situation in which the particular game places its players. It all helps develop a feel for the mathematics of the game and the problems the players face.

Some representative games of evolutionary game theory are hawk-dove, war of attrition, stag hunt, producer-scrounger, tragedy of the commons, and prisoner's dilemma. Some of the various strategies that apply in these games are Hawk, Dove, Bourgeois, Prober, Defector, Assessor, and Retaliator. Depending on the particular “Game” the various strategies vie against one another under the particular game rules, and the mathematics of the evolutionary game theory are used to determine the results and behaviours.

Hawk Dove

Solution for V=2, C=10 and fitness starting base B=4. The fitness of a Hawk for different population mixes is plotted as a black line, that of Dove in red. An ESS (a stationary point) will exist when Hawk and Dove fitness are equal: Hawks are 20% of population and Doves are 80% of the population.

The most classic game (and Maynard Smith's starting point) is the Hawk Dove game. The game was conceived to analyse the animal contest problem highlighted by Lorenz and Tinbergen. It is a contest over a shareable resource. The contestants can be either a Hawk or a Dove. These are not two separate species of bird; they are two subtypes of one species with two different types of strategy (two different morphs). The term Hawk Dove was coined by Maynard Smith because he did his work during the Vietnam War when political views fell into one of these two camps. The strategy of the Hawk (a fighter strategy) is to first display aggression, then escalate into a fight until he either wins or is injured. The strategy of the Dove (fight avoider) is to first display aggression but if faced with major escalation by an opponent to run for safety. If not faced with this level of escalation the Dove will attempt to share the resource.

Payoff Matrix for Hawk Dove Game

	meets Hawk	meets Dove
if Hawk	V/2 - C/2	V
if Dove	0	V/2

Given that the resource is given the value V, the damage from losing a fight is given cost C:

• If a Hawk meets a Dove he gets the full resource V to himself
• If a Hawk meets a Hawk – half the time he wins, half the time he loses…so his average outcome is then V/2 minus C/2
• If a Dove meets a Hawk he will back off and get nothing - 0
• If a Dove meets a Dove both share the resource and get V/2

The actual payoff however depends on the probability of meeting a Hawk or Dove, which in turn is a representation of the percentage of Hawks and Doves in the population when a particular contest takes place. But that population makeup in turn is determined by the results of all of the previous contests before the present contest- it is a continuous iterative process where the resultant population of the previous contest becomes the input population to the next contest. If the cost of losing C is greater than the value of winning V (the normal situation in the natural world) the mathematics ends in an ESS – an evolutionarily stable strategy situation having a mix of the two strategies where the population of Hawks is V/C. The population will progress back to this equilibrium point if any new Hawks or Doves make a temporary perturbation in the population. The solution of the Hawk Dove Game explains why most animal contests involve only “ritual fighting behaviours” in contests rather than outright battles. The result does not at all depend on “good of the species” behaviours as suggested by Lorenz, but solely on the implication of actions of “selfish genes”.

The War of Attrition game

In the Hawk Dove game the resource is sharable, which gives payoffs to both Doves meeting in a pairwise contest. In the case where the resource is not sharable but an alternative resource might be available by backing off and trying elsewhere, pure Hawk or Dove strategies become less effective. If an unshareable resource is combined with a high cost of losing a contest (injury or possible death) both Hawk and Dove payoffs are then further diminished. A safer strategy of lower cost display, bluffing and waiting to win, then becomes viable – a Bluffer strategy. The game then becomes one of accumulating costs, either the costs of displaying or costs of prolonged unresolved engagement. It’s effectively an auction; the winner is the contestant who will swallow the greater cost while the loser gets, for all his pains, the same cost as the winner but NO resource. The resulting evolutionary game theory mathematics leads to an optimal strategy of timed bluffing.

War of attrition for different values of resource. Note the time it takes for an accumulation of 50% of the contestants to quit vs. the Value(V) of resource contested for

This is because in the war of attrition any strategy that is unwavering and predictable is unstable – unstable in the sense that such a strategy will ultimately be displaced by a mutant strategy which will simply rely on the fact that it can best the existing predictable strategy just by investing just an extra small delta of waiting resource to insure that it wins. Therefore, only a random unpredictable strategy can maintain itself in a population of Bluffers. The contestants in effect choose an “acceptable cost” to be incurred related to the value of the resource being sought – randomly selected at contest start - effectively making a related random “bid” which becomes part of a mixed strategy (a strategy where a contestant has several, or even many, possible actions in his strategy). This implements a distribution of bids for a resource of specific value V, where the particular bid made for any specific contest is chosen at random from within that distribution. The distribution (an ESS) can be computed by invoking the Bishop-Cannings theorem, which holds true for any case of mixed strategy ESS. The distribution function in these contests was thus determined by Parker and Thompson to be:

$p(x)={\frac {e^{-x/V}}{V}}.$

This leads to the result where the cumulative population of quitters of the contest versus for any particular cost m in this "mixed strategy" solution is:

$p(m)=1-e^{-m/V},$

which is shown in the adjacent graph. The intuitive sense that greater values of resource sought leads to greater waiting times is borne out here.

The Dung Fly (Scatophaga stercoraria) - a War of Attrition player

The mantis shrimp guarding its home playing the Bourgeois Strategy

Animal Strategy Examples: by examining the behaviours, then determining both the Costs and the Value of resources attained in a contest the strategy of an organism can be verified

This is exactly what is observed in nature for contests in a number of species, for example between male dung flies contesting for mating sites. The timing of disengagement in these contests follows the exact mathematical curve derived from the evolutionary theory mathematics.

Asymmetries that allow new strategies

In the War of Attrition there must be nothing that signals the size of a bid to an opponent, otherwise this can act as a cue which can be utilised by an opponent for an effective counter-strategy for a mutant. There is however a mutant strategy which can better a Bluffer in the War of Attrition Game if such an asymmetry exists. This mutant strategy is the Bourgeois strategy (Maynard Smith named the strategy Bourgeois, because with his background of communism he regarded it as “politically bourgeois” way to value ownership). Bourgeois uses an asymmetry of some sort to break the deadlock. In nature one such asymmetry is possession - which contestant has the prior possession of the resource. The strategy is to play a Hawk if in possession of the resource but to display then retreat if not in possession. This strategy requires greater cognitive capability than Hawk but Bourgeois is still a very prevalent strategy in many animal contests… some examples being mantis shrimp and speckled wood butterfly contests. However, studies done on contests between speckled wood butterflies have been criticized for the methods used. Butterflies that were initially resource owners were held away from their territories very briefly in nets, a method that is suggested to tax the owners' metabolisms and potentially cause them to lose ensuing contests. Thus, evolutionary game theory explains a very wide range of rather mystifying behaviours in animals and in plant contests for many species with an illuminating clarity.

Strategic alternatives in social behaviour

Alternatives for game theoretic social interaction

Games like Hawk Dove and War of Attrition represent pure competition between individuals and have no attendant social element to the game. Where social influences apply there are four possible alternatives for strategic interaction that exist for the competitors. This is shown on the adjacent figure, where a plus sign represents a benefit and a minus sign represents a cost.

• In a Cooperative or Mutualistic relationship both "donor" and "recipient" are almost indistinguishable as both gain a benefit in the game by co-operating, i.e. the pair are in a game-wise situation where both can gain by executing a certain strategy, or alternatively both must act in concert because of some encompassing constraints that effectively puts them "in the same boat".
• In an Altruistic relationship the donor, at a cost to himself provides a benefit to the recipient. In the general case the recipient will have a kin relationship to the donor and the donation is one-way. Behaviours where benefits are donated alternatively (in both directions) at a cost, are often called altruistic, but on analysis such "altruism" can be seen to arise from optimised "selfish" strategies
• Spite is essentially a “reversed” form of altruism where an ally is aided by damaging the ally’s competitor(s). The general case is that the ally is kin related and the benefit is an easier competitive environment for the ally. Note: George Price, one of the early mathematical modellers of both altruism and spite, found this equivalence particularly disturbing at an emotional level.
• Selfishness is the base criteria of all strategic choice from a game theory perspective – strategies not aimed at self-survival and self-replication are not long for any game. Critically however, this situation is impacted by the fact that competition is taking place on multiple levels - i.e. at a genetic, an individual and a group level.

The rationale and the mathematics that lie behind adopting one of these alternative social strategies will be covered in the following sections of this article.

Who is playing the game?

The Belding's ground squirrel lives in communities of closely related females and their young and male “immigrants”. This is so because males leave the colony on reaching maturity and find other colonies to join. When predators are in the vicinity of a colony certain squirrels emit a loud piercing alarm call, allowing other colony members to take cover. This call substantially endangers the caller as it easily locates it for the predator. However, as female squirrels are so closely related evolutionary game theory utilising measures of Inclusive Fitness shows that this behaviour is superior to not calling for them. Field studies confirm this is exactly how the females behave. The males, however, having no such level of inclusive fitness, in general do not call.

At first glance it may be surmised that the contestants of evolutionary games are the individuals present in each generation who directly participate in the game. On reflection, however, we see that individuals live only through one game cycle, and instead it is the strategies that really contest with one another over the duration of these recursive games. So it is ultimately genes that play out a full contest – genes of STRATEGY (i.e. selfish genes). The contesting genes are not just present in an individual and his/her direct linage; they are also present to a relational degree in all of the individual’s kin. This can sometime profoundly affect the kinds of strategies that will survive, and nowhere is this more pertinent than in issues of cooperation and defection. William Hamilton, whose work defined kin selection, was involved in much of the groundbreaking mathematical work in this area and Hamilton treated many of these cases using game theoretic models. Kin related treatment of game contests help to explain many aspects of the behaviours of eusocial insects, the altruistic behaviour in parent/offspring interactions, mutual protection behaviours, and co-operative care of offspring. For such games Hamilton defined an extended form of fitness that is operative – Inclusive fitness, which extends the fitness measure to include an individual’s offspring as well as any other “offspring equivalents” found in kin.

The Mathematics of Kin Selection

The concept of Kin Selection can be formulated into a mathematical relationship by expressing the basic definition that: inclusive fitness = own contribution to fitness + contribution of all relatives. Fitness is measured relative to the average population at large; for example, fitness = 1 means growth at the same average rate of the population, fitness < 1 means decreasing percentage-wise in the population (dying out), fitness > 1 means increasing percentage-wise in the population (taking over). The inclusive fitness of an individual wi is the sum of its specific fitness of itself ai plus the specific fitness of each and every relative weighted by the degree of relatedness which equates to the summation of all rj*bj....... where rj is relatedness of a specific relative and bj is that specific relative's fitness - producing: $w_{i}=a_{i}+\sum _{j}r_{j}b_{j}.$ Now if individual ai sacrifices his "own average equivalent fitness of 1" by accepting a fitness cost C, and then to "get that loss back", wi must still be 1 (or greater than 1)...AND if we use R*B to represent the summation we get: 1< (1-C)+RB ....or rearranging..... R>C/B.

Hamilton then went on beyond areas of kin relatedness to work with Robert Axelrod to analyze games that involved co-operation under conditions not involving kin where reciprocal altruism comes into play.

Meat ant - Iridomyrmex purpureus -Relatedness of a worker ant (always a female) To mother or father=.5 To sister+=.75 To own daughter or son=.5 To brother=.25....... Therefore, it is more advantageous genetically to help produce a sister (.75) than to have a child oneself (.5). Furthermore, since ALL females in colony are sisters, this insures that new colonies that disperse from the existing one will also have strong kin related linkages - further multiplying the benefit to cost ratio of altruism

Eusociality and kin selection

“Ants are good citizens, they place group interests first” _{Clarence Day}

Insect societies have always been a source of fascination and inscrutability. Certainly one of the very most inexplicable behaviours of these eusocial insects is the forfeiture of reproductive rights that the workers grant in favour of the queen. In a Darwinian sense, no greater sacrifice can ever exist. The explanation that has been forthcoming is in Kin Selection influences that arise from the genetic makeup of these workers which predisposes them to such kin oriented altruistic behaviours . Most eusocial insect societies have haplo-diploid sexual determination, which in essence means that males develop from unfertilised eggs, females from fertilised. This leads to the situation in these Haplodiploid species, that sisters share 75% of their genes in common…. in effect more than they genetically share with their mother. But in this system ALL the other workers are sisters, further skewing the kin related payoffs toward altruistic sacrifice. Helping to raise the queen's offspring, which are also sisters, therefore has major payoffs as does sacrificing onself if the nest is invaded by some predator. For those insect societies which do not exhibit haplodiplodal genetic relationships, the most significant example being termites, there still exist kin related influences arising from inbreeding, monogamy and the ineffectiveness of dispersal

The widely accepted kin related explanation of insect eusociality has however been challenged recently by a few highly noted evolutionary game theorists (Nowak and Wilson) who have published a controversial alternative game theoretic explanation based on a sequential development and group selection effects proposed for these insect species.

Prisoners dilemma

One of the great difficulties of Darwinian Theory, and one recognised by Darwin himself was the problem of altruism. If the basis for selection is at the individual level, altruism makes no sense at all. But universal selection at the group level (for the good of the species, not the individual) fails to pass the test of the mathematics of game theory and is certainly not found to be the general case in nature. Yet in many social animals altruistic behaviour can be found. The solution to this paradox can be found in the application of evolutionary game theory to the “Prisoners Dilemma” game - a game which tests the payoffs of cooperating or in defecting from cooperation. It is certainly the most studied game in all of Game Theory.

As with all games in evolutionary game theory the analysis of Prisoners Dilemma is as a repetitive game. This repetitive nature affords competitors the possibility of retaliating for “bad behaviour” (defection) in previous rounds of the game. There is a multitude of strategies which have been tested by the mathematics of EGT and in computer simulations of contests and the conclusion is that the best competitive strategies are general cooperation with a reserved retaliatory response if necessary. The most famous and certainly one of the most successful of these strategies is Tit for Tat which carries out this approach by executing a simple algorithm.

procedure tit-for-tat
EventBit:=Trust;

do while Contest=ON;
    if Eventbit=Trust then
        Cooperate 
    else
        Defect;
    
    if Opponent_Move=Cooperate then 
        EventBit:=Trust 
    else 
        Eventbit:=NOT(Trust);
end;

The pay-off for any single round of the game is defined by the pay-off matrix for a single round game (shown in bar chart 1 below). In multi-round games the different choices - Co-operate or Defect - can be made in any particular round, resulting in a certain round payoff. It is, however, the possible accumulated pay-offs over the multiple rounds that count in shaping the overall pay-offs for differing multi-round strategies such as Tit-for-Tat.

Two bar charts showing the payoffs for various actions in two varieties of Prisoners Dilemma game. The Prisoners Dilemma situation - Co-operate or Defect?: Two robbers that have promised never to “snitch” on each other are arrested, but police don’t have enough evidence for a maximum sentence. If the robbers both stick to their promise and don’t snitch on each other the police can only charge and convict them both with a lesser sentence. The robbers are separated and interrogated in different rooms where both are offered the chance at “turning state’s evidence” - escaping completely any sentence for himself, but sticking his partner with the maximal sentence . Each worries that even if he keeps his pledge, his partner will snitch and make him a total sucker for being so true to his word and landing himself a maximum sentence. The characteristic of the payoffs for Prisoners Dilemma in any individual round of play are: Payoff _{(Temptation in Defecting vs. Co-operation)} > Payoff _{(Mutual Co-operation)} > Payoff_{(Joint Defection)} > Payoff _{(Sucker - you co-operate but opponent defects)}

Example 1: This is the straightforward single round Prisoners Dilemma Game (the game is executed in only one round). The classic Prisoners Dilemma game payoffs gives you a maximum payoff if you defect and your "partner" co-operates (this choice for you is known as TEMPTATION). If however you co-operate and your partner defects you get the worst possible result ( the SUCKERS PAYOFF ). In these payoff conditions the best choice (a Nash Equilibrium in classic game theory) is to defect.

Example 2: This is Prisoners Dilemma played repeatedly. The strategy employed is Tit-for-Tat which alters behaviors based on the action taken by a partner in the previous round - i.e. reward co-operation and punish defection. The effect of this strategy in ACCUMULATED payoff over many rounds is to produce a higher payoff for both players co-operation and a lower payoff for defection. This removes the Temptation to defect. The suckers payoff also becomes less, although "invasion" by a pure defection strategy is not entirely eliminated (this can be overcome by clustering of cooperative strategies - see spatial games below).

Routes to altruism

Altruism takes place when one individual, at a cost C to itself, exercises a strategy that provides a benefit B to another individual. The cost may consist of a loss of capability or resource which helps in the “battle for survival and reproduction” or it may consist of an added risk to its own survival – i.e. incurring a higher probability of non-survival for itself. Altruism strategies arise in a number of ways:

Type	Applies to:	Situation	Mathematical effect
Kin Selection – (inclusive fitness of related contestants)	Kin – genetically related individuals	Evolutionary Game participants are actually “genes of strategy”. So it follows that what is the best payoff for an individual is not necessarily what is the best payoff for the gene. In any generation the player gene is NOT only in one individual, it is in a Kin-Group. What produces the highest fitness payoff for the Kin Group is the better strategy and will be selected by natural selection. Therefore, strategies that include “self-sacrifice” on the part of individuals are often the “game winner” – the evolutionarily stable strategy. Of course animals must necessarily live in kin-group during some life phase or over all of the game to afford the opportunity for this altruistic sacrifice to ever take place.	Games must take into account Inclusive Fitness. Fitness function is the combined fitness of a group of related contestants – each weighted by the degree of relatedness – relative to the total genetic population. The mathematical analysis of this gene centric view of the game leads to "Hamilton's rule" that the genetic relatedness of the altruistic donor must exceed the cost-benefit ratio of the altruistic act itself, in mathematical terms: R>c/b R is relatedness, c the cost, b the benefit
Direct reciprocity	Related or non related contestants that “trade favours” in paired relationships	A game theoretic embodiment of the adage “I’ll scratch your back if you scratch mine”. A pair of individuals exchange favours in a multi-round game in which both individuals are involved. The individuals in the pair are specifically recognisable to one another as partnered. The term “DIRECT” applies because the return favour is specifically given back to the pair partner only.	The characteristics of the multi-round game produce a danger of “defection” and the potentially lesser payoffs of “cooperation” in each round, however any such defection can lead to subsequent "punishment" in a following round – establishing the game as that of classical repeated Prisoners Dilemma. Therefore, the family of tit-for-tat strategies come to the fore.
Indirect Reciprocity	Related or non related contestants “trade favours” but not with partnering involved. A return favour is “implied” but with no specific identified source who is to give it.	This behaviour is akin to “I’ll scratch your back, you scratch someone elses back, another someone else will scratch mine (probably)". The return favour is not derived from any particular established partner. The potential for indirect reciprocity exists for a specific organism if it lives in a cluster of individuals who can interact over an extended period of time. It has been argued that human behaviours in establishing moral system as well as the expending of significant energies in human society for tracking individual reputation is a direct effect of societies reliance on strategies of indirect reciprocation.	The game is highly susceptible to defection, particularly as direct retaliation becomes essentially impossible. Therefore, indirect reciprocity will not work mathematically unless some additional factor is at play to verify that any individual being granted a favour is “trustworthy” to return the favour in the sharing group and insuring the punishment (non-reward) of defectors for “misbehaving”. This requires essentially keeping a "social score", a measure of the past co-operative behaviour. The mathematics leads to a modified version of Hamilton's Rule where: q>c/b where q (the probability of knowing the social score) must be greater than the cost benefit ratio Organisms that use this sort of social score as an extended modification of the tit-for-tat variant strategies are termed Discriminators. Different discrimination algorithms exist for determining social scores, but in general all of these require a much higher level of cognition to exist than strategies of simple direct reciprocity. As David Haig, a noted evolutionary biologist succinctly put it - "For direct reciprocity you need a face; for indirect reciprocity you need a name".

The ESS

The Evolutionarily Stable Strategy (ESS) is perhaps the most widely known albeit most widely misunderstood concept in evolutionary game theory. The ESS is basically akin to Nash Equilibrium in classical Game Theory, but with mathematically extended criteria.

The Payoff Matrix for the Hawk Dove Game with an additional strategy - the Assessor Strategy. The Assessor competitor "studies its opponent" and behaves as a Hawk when matched with an opponent it judges "weaker", and behaves like a Dove when it assesses its opponent as stronger. (The usual judgement criteria in nature is opponent size) Assessor is an ESS. As can be seen in the payoffs of the matrix it can invade as a mutant into both Hawk and Dove populations, and can ALSO withstand invasion by either Hawk or Dove mutants as it gets a better payoff when matched against another Assessor

Nash Equilibrium is a game equilibrium where it is not rational for any player to deviate from the present strategy they are executing. As discussed, in Evolutionary game Theory contestants are NOT behaving with rational choice, nor do they have the ability to totally alter their strategy, aside from executing a very limited “mixed strategy”. An ESS is instead a state of game dynamics where, in a very large (or infinite) population of competitors, another mutant strategy cannot successfully enter the population to disturb the existing dynamic (which in itself is population mix dependent). This leads to a situation where to be a successful strategy having an ESS, the strategy must be both effective against competitors when it is rare - to enter the previous competing population, and also successful when later in high proportion in the population - to “defend itself”. This in turn necessarily means that the strategy needs to be successful when it contends with others exactly like itself. ESS is NOT:

• An OPTIMAL strategy – an optimal strategy would maximize Fitness, and many ESS states are far below the maximum fitness achievable in a fitness landscape. (see Hawk Dove graph above as an example of this)
• A singular solution – often several ESS conditions can exist in a competitive situation. A particular contest might stabilize into any one of these possibilities, but later a major perturbation in conditions can move the solution into one of the alternative ESS states.
• Always present - It is also possible for there to be no ESS. An example evolutionary game with no ESS is the Rock-Scissors-Paper game found in a number of species (an example the side-blotched lizard (Uta stansburiana))
• An unbeatable strategy - The ESS strategy is not necessarily an unbeatable strategy, it is only an uninvadable one.

An Assessor Strategy Player - - Female funnel web spiders (Agelenopsis aperta) contest with one another for the possession of their desert spider webs. A higher-quality established web location offers a considerable reproductive advantage and is well worth holding, but serious fights for them are potentially very dangerous for the contestants. The spiders have been found to use the Assessor strategy, with size the critical determinant of winner. When an intruder spider enters a web to contest for possession the spiders become involved in a “web bouncing” behaviour which in essence establishes size by "weighing" the two contestants. The smaller spider, even if the original possessor of the web, leaves the site of the contest. In a detailed study Riechert proved that an Assessor "weighing" strategy was in play by adding a ballast weight to physically smaller spiders.

The ESS state can be solved for mathematically by exploring either the dynamics of population change to determine any ESS.... or alternatively by solving equations for the stable stationary point conditions which fundamentally define an ESS. For example, in the Hawk Dove Game we can look for whether there is a static population mix condition where the fitness of Doves will be exactly the same as fitness of Hawks (therefore both having equivalent growth rates - a "static point").

Let chance of meeting a Hawk=p so therefore the chance of meeting a dove is (1-p)

Let WHawk equal the Payoff for Hawk.....

WHawk=Payoff in the chance of meeting a Dove + Payoff in the chance of meeting a Hawk

Taking the PAYOFF MATRIX results and plugging them into the above equation:

WHawk= V·(1-p)+(V/2-C/2)·p

Similarly for a Dove:

WDove= V/2·(1-p)+0·(p)

so....

WDove= V/2·(1-p)

Equating the two fitnesses, Hawk and Dove

V·(1-p)+(V/2-C/2)·p= V/2·(1-p)

... and solving for p

p= V/C

so for this "static point" where the Population Percent is an ESS solves to be ESS_{(percent Hawk)}=V/C

Similarly using inequalities it can be shown that an additional Hawk or Dove “mutant” entering this ESS state generates a situation leading eventually to LESS fitness for their kind – both a true Nash and an ESS equilibrium. This fairly simple example shows that when the risks of contest injury or death (the Cost C) is significantly greater than the potential reward offered (the benefit value V) then the stable population which is reached will be MIXED between the aggressors and the doves, and that the proportion of doves will exceed that of the aggressors. This then mathematically explains behaviours that are actually observed in nature.

Rock Paper Scissors game

Rock Paper Scissors

Mutant Invasion for Rock Paper Scissors payoff matrix - an endless cycle

An evolutionary game that actually turns out to be a children’s game is rock-paper-scissors. The game is simple – rock beats scissors (blunts it), scissors beats paper (cuts it), and paper beats rock (wraps it up). Anyone who has ever played this simple game knows that it is not sensible to have any favoured play – your opponent will soon notice this and switch to the winning counter-play. The best strategy (a Nash equilibrium) is to play a mixed random game with any of the three plays taken a third of the time. This, in EGT terms, is a mixed strategy. But many lifeforms are incapable of mixed behavior — they only exhibit one strategy (known as a “pure” strategy”). If the game is played only with the pure Rock, Paper and Scissors strategies the evolutionary game is dynamically unstable: Rock mutants can enter an all scissor population, but then – Paper mutants can take over an all Rock population, but then – Scissor mutants can take over an all Paper population – and on and on…. This is easily seen on the game payoff matrix, where if the paths of mutant invasion are noted, it can be seen that the mutant "invasion paths" form into a loop. This in triggers a cyclic invasion behaviour.

Rock Paper Scissors Dynamics A computer simulation of the Rock Scissors Paper evolutionary game. The associated RPS Game Payoff Matrix is shown. Starting with an arbitrary population the percentage of the three morphs builds up into a continuously cycling pattern – with rock leading paper, paper leading scissors, and scissors leading rock etc. etc.

The side-blotched lizard

The side-blotched lizard (Uta stansburiana) is polymorphic with three morphs that each pursues a different mating strategy

Uta stansburiana — the side-blotched lizard

1) The orange throat is very aggressive and operates over a large territory - attempting to mate with numerous females within this larger area

2) The unaggressive yellow throat (called “sneakers”) mimic the markings/behavior of female lizards and sneakily slip into the orange throat's territory to mate with the females there (thereby overtaking the population), and

3) The blue throat who mates with and carefully guards ONE female - making it impossible for the sneakers to succeed and therefore overtakes their place in a population…

However the blue throats cannot overcome the more aggressive orange throats… And on, and on… The overall situation corresponds to the exact form of the Rock, Scissors, Paper game and the dynamics is much the same. The populations for these lizards actually cycle on a six-year basis. Once again EGT explains a very curious and otherwise inexplicable behaviour in the field of biology. When he read that lizards of the species Uta stansburia were essentially engaged in a game with rock-paper-scissors structure John Maynard Smith exclaimed `They have read my book!´

RPS and ecology

Rock Paper Scissors, incorporated into an evolutionary game has been used for modeling natural processes in the study of Ecology. In two-strategy “pairwise” contests, usually one strategy becomes dominant then maintains its dominance, bringing the system to a static ESS point. Interpreted as modelling the competition between species in nature, this suggests that one species will tend to dominate a population of organisms occupying a single ecological niche. However, most ecologies support many species, each dominating its own niche. With more than two species present, mechanisms come into play which bring more complex dynamics to these systems. For example, competition occurring now between two of the species might indirectly help a third by weakening a future rival before direct competition takes place. In such cases, the ESS point can take on a more complex character; even developing a distribution of species which is cyclic over time. The Rock Paper Scissors game was used as the basis for modelling these multi-species ecologies, producing dynamics closely matching observations from nature.

RPS and human social cyclic behavior

Using experimental economics methods, scientists have used RPS game to test the human social evolutionary dynamical behaviors in laboratory. The social cyclic behaviors, predicted by evolutionary game theory, have been also observed in the laboratory experiments with various protocol. These findings mean that evolutionary game theory can be a powerful framework for social science as well as for biology and ecology.

Signalling, sexual selection and the handicap principle

The Handicap Principle in action

Aside from the difficulty of explaining how altruism exists in many evolved organisms Darwin was also bothered by a second conundrum – why do a significant number of species have phenotypical attributes that are patently disadvantageous to them with respect to their survival - and should by the process of natural section be selected against – e.g. the massive inconvenient feather structure found in a peacock's tail? Regarding this issue Darwin wrote to a colleague “ The sight of a feather in a peacocks tail, whenever I gaze at it, makes me sick”. It is the mathematics of evolutionary game theory, which has not only explained the existence of altruism but also explains the totally counterintuitive existence of the peacock’s tail and other such biological encumbrances.

On analysis, problems of biological life are not at all unlike the problems that define economics – eating (akin to resource acquisition and management), survival (competitive strategy) and reproduction (investment, risk and return). Game theory was originally conceived as a mathematical analysis of economic processes and indeed this is why it has proven so useful in explaining so many biological behaviours. One important further refinement of the EGT model that has particular economic overtones rests on the analysis of COSTS. A simple model of cost assumes that all competitors suffer the same penalty imposed by the Game costs, but this is not the case. More successful players will be endowed with or will have accumulated a higher “wealth reserve” or “affordability” than less successful players. This wealth effect in evolutionary game theory is represented mathematically by “resource holding potential (RHP)” and shows that the effective cost to a competitor with higher RHP are not as great as for a competitor with a lower RHP. As a higher RHP individual is more desirable mate in producing potentially successful offspring, it is only logical that with sexual selection RHP should have evolved to be signalled in some way by the competing rivals, and for this to work this signalling must be done honestly. Amotz Zahavi has developed this thinking in what is known as the Handicap Principle, where superior competitors signal their superiority by a costly display. As higher RHP individuals can properly afford such a costly display this signalling is inherently honest, and can be taken as such by the signal receiver. Nowhere in nature is this better illustrated than in the magnificent and costly plumage of the peacock. The mathematical proof of the handicap principle was developed by Alan Grafen using evolutionary game-theoretic modelling.

Co-evolution

Two types of dynamics have been discussed so far in this article:

• Evolutionary games which lead to a stable situation or point of stasis for contending strategies which result in an evolutionarily stable strategy
• Evolutionary games which exhibit a cyclic behaviour (as with RPS game) where the proportions of contending strategies continuously cycle over time within the overall population

Competitive Co-evolution - The rough-skinned newt (Tarricha granulosa) has enough poison in its body to kill a human being 30 times over. This excess of lethal capability is due to the fact that it is involved in an arms race with a specific predator, the common garter snake (Thamnophis sirtalis), which in response to the newt's poisonous defenses has over time evolved to be highly tolerant of the poison. The two species are thereby locked in a "Red Queen" arms race.

Mutualistic Coevolution - Darwin's orchid (Angraecum sesquipedale) and the moth Morgan's sphinx (Xanthopan morgani) like many insect and flower "partners" have a mutual relationship where the moth gains pollen and the flower pollination. To insure efficiency in this particular exchange the pair have evolved a mechanism which excludes the waste of pollen transfer to/from another flower species and assures feeding pollen only to the "proper" pollinator. The mechanism is an extraordinarily long proboscis on the moth and a equally long nectary on the orchid.

A third dynamic can exist in certain more complex systems, where combinations of effects are in play, which contain not only intra-species competition but also inter-species competition as well. This is the realm of co-evolution. Examples include predator-prey competition and host-parasite competition. There are also co-evolutionary situations possible where two species interact where mutual interests are at play in the dynamic. Evolutionary game models have been created to encompass both these co-evolutionary situations for pairwise and multi-species coevolutionary systems. The general dynamic differs between competitive systems and mutualistic systems.

In competitive (non-mutualistic) inter-species coevolutionary system the species are involved in an “arms race” - where adaptions that are better at competing against the other species are less likely to be culled. Both game payoffs and replicator dynamics are thereby affected to reflect this. The “counterstrategy” of the other competitor species is similarly affected, exposing them to a positive selection pressure for any effective counter-strategy – in effect creating an overall competitive arms race. Underlying this total situation is an ecological effect in which the population dynamic of the competing species itself influences outcomes- e.g. a mutant strategy in a predator population which enables that species to drive its sole prey species to extinction is itself doomed. This all leads in effect to a “Red Queen” dynamic where, as in Alice in Wonderland, the protagonists must “run as fast as they can to just stay in one place”.

A number of EGT models have been produced to encompass coevolutionary situations. Modelling these multi-component coevolutionary systems mathematically is necessarily complex. This is particularly true, as a key factor applicable in these coevolutionary systems is the continuous adaption of strategy taking place in the dynamics of such arms races. Most other evolutionary games, e.g. resource conflicts, do not need to include this process of dynamic mutation in the analysis of game dynamics itself. Therefore, coevolutionary modelling often involves a cross-over area, where both EGT and Genetic Algorithms reflecting mutational effects are used in the models, often having computers simulate the dynamics of the overall coevolutionary game. The resulting dynamics are studied as various parameters are modified. Because a multitude of variables are simultaneously at play in these complex dynamics, solutions become the province of multi-variable optimality, rather than simply a single ESS. The mathematical criteria of determining stable points are Pareto efficiency and Pareto dominance, which is a measure of solution optimality peaks in these interdependent multivariable systems.

In a paper by Carl Bergstrom and Michael Lachmann, the authors successfully apply evolutionary game theory models to understand the division of benefits in mutualistic interactions between organisms. Darwinian assumptions about fitness are modeled using replicator dynamics to show that the organism evolving at a slower rate in a mutualistic relationship will gain a disproportionately high share of the benefits or payoffs. This application of EGT provided an interesting and perhaps unexpected twist on the Red Queen Hypothesis which concludes evolution favored faster rates of evolution.

Extensions of the evolutionary game theory model

A mathematical model analysing the behaviour of a system needs initially to be as simple as possible to aid in developing a base understanding the fundamentals, or “first order effects”, pertaining to what is being studied. With this understanding in place it is then appropriate to see if other, more subtle, parameters (second order effects) further impact the primary behaviours or shape additional behaviours in the system. Following Maynard Smith’s seminal work in EGT, the subject has had a number of very significant extensions which have shed more light on understanding evolutionary dynamics, particularly in the area of altruistic behaviors. Some of these key extensions to EGC are:

A Spatial Game
In a spatial evolutionary game contestants meet in contests at fixed grid positions and only interact with immediate neighbors. Shown here are the dynamics of a Hawk Dove contest, showing Hawk and Dove contestants as well as the changes of strategy taking place in the various cells

Spatial Games	There are several very important influences attributable to geographic factors in the field of evolution – not least of which is gene flow and horizontal gene transfer. Spatial game models take geometry into effect by locating contestants in a lattice of cells over a two dimensional plane to represent this geographic element. In spatial games contests take place only with immediate neighbours. Winning strategies take over these immediate neighbourhoods and then go on to interact further with adjacent neighbourhoods. This model has been especially useful in showing how pockets of co-operators can invade and introduce altruism in the Prisoners Dilemma game, where for example Tit for Tat (TFT) is a Nash Equilibrium but NOT also an ESS. TFT cannot normally invade an all Defector population yet in spatial games this becomes possible. The basic principle of spatial structure is sometimes abstracted into a general network of interactions. This is the foundation of evolutionary graph theory.
Effects of having information	In conventional Game Theory the effect of Signalling (the acquisition of information) is of critical importance in outcomes. This can also be the case in EGT. Nowhere is this more important than in the area of studying Indirect Reciprocity in Prisoners Dilemma (where contests between the SAME paired individuals is NOT repetitive). This models the reality of most normal social interactions which are non-kin related. It has been shown that unless a probability measure of REPUTATION is available in Prisoners Dilemma only DIRECT reciprocity can be achievable. With this information INDIRECT reciprocity is also supported. Alternatively, agents might have access to an arbitrary signal that is initially uncorrelated to strategy but becomes correlated due to evolutionary dynamics. This is known as the green-beard effect or evolution of ethnocentrism in humans. Depending on the game, It can allow the evolution of either cooperation or irrational hostility. More generally, there are many instances in biology from the molecular to the multicellular levels where a signaling game model, with information asymmetry between sender and receiver, might be appropriate, such as in mate attraction or evolution of translation machinery from RNA strings.
Finite populations	Many evolutionary games have been modelled in finite populations rather than infinite to see the effect this may have. Though in many cases this does not significantly alter game dynamics in others significant differences can and do occur, for example being in the prevalence of mixed strategies.

Further application

The successful application of game theory to evolution has brought further insights to human behavior. Whereas game theory traditionally assumes rational actors, in the real world this does not always describe human behavior. EGT has predicted behaviors in animals where strong assumptions of rationality cannot be made. An important feature of all these sets of models under the umbrella of evolutionary game theory is repetition. If the games were not repeated, these EGT models would not be able to provide any insight into adaptive behaviors and strategies due to the dynamic nature of the mechanisms of evolution. Further, this biological application is meaningful for economics because it provides an understanding of the adjustments that occur between two equilibria (Samuelson 2002). While game theory provides a framework within which biologists can learn and understand organisms, the observation of evolution and how these strategies are applied helps economics illuminate processes. It is interesting to note however, that Maynard Smith himself never felt that the use of evolutionary game theory in the field of economics was quite relevant. In a new introduction for his book "The Alchemy of Finance", George Soros says: "While I still cannot claim full familiarity with the literature, I see some merit in behavioral economics and even more merit in evolutionary game theory. I consider the latter a practical exploration of reflexivity." Moreover,the game theory models of the pathogenic microbes may provide us with advanced therapeutic ways.

See also

• Adaptive dynamics
• Behavioral ecology
• Dynamical systems
• Evolution and the Theory of Games (book)
• Evolutionarily stable strategy
• Gene-centered view of evolution
• Memetics
• Evolutionary dynamics

Bibliography

• Maynard Smith, J. (1982) Evolution and the Theory of Games.
• P. Hammerstein and R. Selten, "Game theory and evolutionary biology", in Handbook of Game Theory with Economic Applications, R. J. Aumann and S. Hart, Eds. (Elsevier, Amsterdam, 1994), vol. 2, pp. 929–993
• Hofbauer, J. and Sigmund, K. (1998) Evolutionary games and population dynamics, Cambridge University Press
• Taylor, P. D. (1979). Evolutionarily Stable Strategies with Two Types of Players J. Appl. Prob. 16, 76-83.
• Taylor, P. D., and Jonker, L. B. (1978). Evolutionarily Stable Strategies and Game Dynamics Math. Biosci. 40, 145-156.
• Web of Stories, Interview - Life and Work - John Maynard Smith by Richard Dawkins, http://www.webofstories.com/play/7301?o=S&srId=240151

Further reading

For the Interested Layperson

• Sigmund, Karl; "Games of Life", Penguin Books, also Oxford University Press, 1993, ISBN 0198547838
• Dawkins, Richard; "The Selfish Gene", Oxford University Press, ISBN 0-19-929114-4
• Davis, Morton,; "Game Theory – A Nontechnical Introduction", Dover Books, ISBN 0-486-29672-5
• Axelrod, Robert; "The Evolution of Co-operation", Penguin Books, 1990, ISBN 0-14-012495-0

Historical perspective

• Kohn, Marek; "A Reason for Everything", Faber and Faber, ISBN 0-571-22393-1
• Harman, Oren; "The Price of Altruism", Bodley Head Books, 2010, ISBN 978-1-84792-062-1
• Segerstrale, Ullica; "Nature's Oracle - The life and work of W.D. Hamilton", Oxford University Press,2013, ISBN 978-0-19-860727-4

For the Serious Reader with Appreciation of Mathematics

• Maynard Smith, John; "Evolution and the Theory of Games", Cambridge University Press, ISBN 0-521-28884-3
• Dugatkin and Reeve; "Game Theory and Animal Behavior", Oxford University Press, ISBN 0-19-513790-6

For the mathematically adept

• Nowak, Martin; "Evolutionary Dynamics", Harvard University Press, ISBN 978-0-674-02338-3
• Vincent and Brown; "Evolutionary Game Theory, Natural Selection and Darwinian Dynamics", Cambridge University Press, ISBN 0-521-84170-4
• Hofbauer and Sigmund; "Evolutionary Games and Population Dynamics", Cambridge University Press, ISBN 0-521-62570-X

External links

• EGT at the Stanford Encyclopedia of Philosophy
• Evolving Artificial Moral Ecologies at The Centre for Applied Ethics, University of British Columbia
• Template:Dmoz