The idea of an evolutionarily stable strategy (or ESS) was introduced by Robert Axelrod in his work described in The Evolution of Cooperation, in the context of strategies used to compete at the iterated prisoner's dilemma.
An ESS depends on the idea of invasion, where a population of strategy-X players is visited by a strategy-Y player. The new player is said to invade if, following strategy Y, he scores better than the average strategy-X player. Assuming players are able to choose and switch strategies, this would induce the indigenous population to start switching to strategy Y. In many cases there are diminishing returns for the later adopters, and what follows is an equilibrium ratio of strategy-X players to strategy-Y players.
A strategy X is evolutionarily stable if there is no strategy Y that can invade it. That is, anybody bringing a new strategy into a population of strategy-X players will fare no better on average than the X players are already doing.