Impact evaluation like cost-benefit analysis start with the premise that any intervention in an economic system has knock-on impacts throughout that system: some significant, many small; some costs, some benefits; some people benefit, others are harmed. They aim to identify costs and benefits throughout the system, then quantify them, so that people do not make unwarranted claims of impact and so that informed decisions can be made. The World Bank and Inter bank in America have developed guidelines, but analysis is considered particularly difficult with Third World agriculture.Peter Griffiths, an economist who has worked in 35 developing countries, has examined the additional problems that arise when analysing the impact of Fairtrade interventions, arising from the way it is organized. Impact studies require study of the counterfactual, using control groups to ensure that an impact is the result of the intervention being studied, which is often impossible and always difficult with Fairtrade because of the way that Fairtrade is organized, notably because the Fairtrade farmers and cooperatives are selected from the richer and more efficient, and because the non-Fairtrade farmers may be harmed by Fairtrade. These problems arise with both case studies and more general surveys, and are particularly intractable when the appropriate control group is in another country. There have been virtually no base period studies, to determine the position before a cooperative joined Fairtrade, so it is seldom possible to claim that the farmers in a cooperative are better off or have more self-esteem because of Fairtrade, rather than because Fairtrade selects efficient cooperatives with educated, efficient, well-off farmers. The possibility that all groups of farmers have increased incomes because of higher world prices, changed exchange rates etc. is difficult to tackle. As most of the extra money paid by consumers is taken by firms in rich countries or is spent on added costs incurred in meeting Fairtrade standards, it is seldom possible to identify direct financial benefit to farmers, and the studies concentrate on non-money benefits, like improvement in self-esteem. These are difficult to measure, and it is particularly difficult to measure these non-money benefits in control groups of local non-Fairtrade farmers, the extent to which non-Fairtrade farmers lose self-esteem, etc. because they are excluded from Fairtrade, for instance, and even more difficult to measure any effects on non-Fairtrade farmers in other, poorer, countries. The decision on which such measures to include introduces a bias: it may be asked whether there are other non-money costs or benefits that might be included in the studies which would give answers less favourable to Fairtrade.
An unpublished consultancy report prepared for Fairtrade  found only 33 studies that met their criteria for impact studies out of everything written on Fairtrade in the previous ten years. These 33 included unpublished undergraduate and masters’ dissertations, unpublished theses, journalistic articles by employees or members of Fairtrade cooperatives, several reports on the same cooperatives, and one report cited under two different titles. Griffiths suggests that even this is optimistic: there are very few studies that examine any of the costs identified by critics of Fairtrade in the Fair trade debate, and fewer still that attempt to quantify them. There are few mentions in the literature of the biggest financial costs and benefits, the extra price paid by consumers and the extra profit made by companies in rich countries, and fewer attempts to quantify these. One impact, the amount of money going to businesses in rich countries rather than to Third World producers, is addressed in only a handful of studies: these show that in some cases 90% to 99.5% of the extra money paid by consumers is kept as extra profit by firms in rich countries but cannot show that this is typical: the conclusion is rather that secretiveness prevents impact analysis of the full Fairtrade system. Few analyse how much extra money is received by the cooperatives, what it is spent on, how much is taken by managers of cooperatives and how much reaches the farmers. Few discuss the costs identified in the Fair trade debate.
There are, however, a lot of case studies of Fairtrade cooperatives which do not attempt to be impact studies, identifying and quantifying all positive and negative impacts, but are, rather, examinations of how subsections of the system operate, usually single Fairtrade cooperatives and sometimes one Fairtrade product in a country - valuable but different research.
Even if twenty or thirty studies which met the impact evaluation criteria could be identified, it would not be possible to generalize from them because nobody claims that they are typical. The studies are not of randomly selected cooperatives: they are concentrated on better known, more successful and well established ones, and there are few which examine the problems of cooperatives which have recently joined Fairtrade or of the least successful quartile of Fairtrade cooperatives.
^Winters, P., Alessandro Maffioli and Lina Salazar, ‘Introduction to the Special Feature: Evaluating the Impact of Agricultural Projects in Developing Countries’ Journal of Agricultural Economics, Vol. 62, No. 2, 2011, 393–402 doi: 10.1111/j.1477-9552.2011.00296.x
^Winters, P., Salazar, L. and Maffioli, A. ‘Designing impact evaluations for agricultural projects’, Impact Evaluation Guidelines, Strategy Development Division, Technical Notes No. IDB-TN-198. (Washington, DC: Inter-American Development Bank, 2010). Accessed at http://www.iadb.org/document.cfm?id=35529432
^Angelucci, M. and Di Maro, V. Project Evaluation and Spillover Effects. Impact Evaluation Guidelines, Strategy Development Division, Technical Notes No. IDB-TN-136 (Washington, DC: Inter-American Development Bank, 2010). Accessed at http://www-personal.umich.edu/
^Michael A. Clemens and Gabriel Demombynes, ‘When Does Rigorous Impact Evaluation Make a Difference? The Case of the Millennium Villages’, World Bank Policy Research Working Paper 5477, 2010
^Valkila, J., Haaparanta, P., & Niemi, N. (2010). “Empowering Coffee Traders? The Coffee Value Chain from Nicaraguan Fair Trade Farmers to Finnish Consumers.” Journal of Business Ethics , 97:257-270
^Kilian, B., Jones, C., Pratt, L., & Villalobos, A. (2006). “Is Sustainable Agriculture a Viable Strategy to Improve Farm Income in Central America? A Case Study on Coffee”. Journal of Business Research , 59(3), 322–330.