Exploratory data analysis

Exploratory data analysis (EDA) is an approach to analysing data for the purpose of formulating hypotheses worth testing, complementing the tools of conventional statistics for testing hypotheses. It was so named by John Tukey to contrast with Confirmatory Data Analysis, the term used for the set of ideas about hypothesis testing, p-values, confidence intervals etc. which formed the key tools in the arsenal of practising statisticians at the time.

EDA development

Tukey held that too much emphasis in statistics was placed on statistical hypothesis testing (confirmatory data analysis); more emphasis needed to be placed on using data to suggest hypotheses to test. In particular, he held that confusing the two types of analyses and employing them on the same set of data can lead to systematic bias owing to the issues inherent in testing hypotheses suggested by the data.

The objectives of EDA are to:

• Suggest hypotheses about the causes of observed phenomena
• Assess assumptions on which statistical inference will be based
• Support the selection of appropriate statistical tools and techniques
• Provide a basis for further data collection through surveys or experiments

Many EDA techniques have been adopted into data mining and are being taught to young students as a way to introduce them to statistical thinking.

Techniques

There are a number of tools that are useful for EDA, but EDA is characterized more by the attitude taken than by particular techniques.

The principal graphical techniques used in EDA are:

• Box plot
• Histogram
• Multi-vari chart
• Run chart
• Pareto chart
• Scatter plot
• Stem-and-leaf plot
• Odds ratio
• Chi-square

The principal quantitative techniques are:

• Median polish
• the Trimean
• Letter values
• Resistant line
• Resistant smooth
• Rootogram

Graphical and quantitative techniques are:

• Multidimensional scaling
• Ordination

History

Many EDA ideas can be traced back to earlier authors, for example:

• Francis Galton emphasized order statistics and quantiles.
• Arthur Bowley used precursors of the stemplot and five-number summary (Bowley actually used a "seven-figure summary", including the extremes, deciles and quartiles, along with the median - see his Elementary Manual of Statistics (3rd edn., 1920), p. 62 - he defines "the maximum and minimum, median, quartiles and two deciles" as the "seven positions").
• Andrew Ehrenberg articulated a philosophy of data reduction (see his book of the same name).

The Open University course Statistics in Society (MDST 242), took the above ideas and merged them with Gottfried Noether's work, which introduced statistical inference via coin-tossing and the median test.

Software

• OpenSHAPA (modern open source successor to MacSHAPA), permits analysis of various media files (e.g. video, sound).
• CMU-DAP (Carnegie-Mellon University Data Analysis Package, FORTRAN source for EDA tools with English-style command syntax, 1977).
• Data Applied, a comprehensive web-based data visualization and data mining environment.
• Fathom (for high-school and intro college courses).
• JMP, an EDA package from SAS Institute.
• KNIME Konstanz Information Miner - Open-Source data exploration platform based on Eclipse.
• LiveGraph (open source real-time data series plotter).
• Orange, an open-source data mining software suite.
• SOCR provides a large number of free Internet-accessible.
• DASS-GUI - data mining framework written in C++ and Qt.
• TinkerPlots (for upper elementary and middle school students).

See also

• Anscombe's quartet, on importance of exploration
• Predictive analytics
• Structured data analysis (statistics)

Bibliography

• Andrienko, N & Andrienko, G (2005) Exploratory Analysis of Spatial and Temporal Data. A Systematic Approach. Springer. ISBN 3-540-25994-5
• Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1985). Exploring Data Tables, Trends and Shapes. ISBN 0-471-09776-4. {{cite book}}: Cite has empty unknown parameter: |coauthors= (help)CS1 maint: multiple names: authors list (link)
• Hoaglin, D C; Mosteller, F & Tukey, John Wilder (Eds) (1983). Understanding Robust and Exploratory Data Analysis. ISBN 0-471-09777-2. {{cite book}}: Cite has empty unknown parameter: |coauthors= (help)CS1 maint: multiple names: authors list (link)
• Leinhardt, G., Leinhardt, S., Exploratory Data Analysis: New Tools for the Analysis of Empirical Data, Review of Research in Education, Vol. 8, 1980 (1980), pp. 85–157.
• Theus, M., Urbanek, S. (2008), Interactive Graphics for Data Analysis: Principles and Examples, CRC Press, Boca Raton, FL, ISBN 978-1-58488-594-8
• Tukey, John Wilder (1977). Exploratory Data Analysis. Addison-Wesley. ISBN 0-201-07616-0. {{cite book}}: Cite has empty unknown parameters: |origmonth=, |month=, |chapterurl=, |origdate=, and |coauthors= (help)
• Velleman, P F & Hoaglin, D C (1981) Applications, Basics and Computing of Exploratory Data Analysis ISBN 0-87150-409-X
• Young, F. W. Valero-Mora, P. and Friendly M. (2006) Visual Statistics: Seeing your data with Dynamic Interactive Graphics. Wiley ISBN 978-0-471-68160-1

External links

• Carnegie Mellon University - free online course on EDA
• University of Minnesota - EDA class notes
• - International Workshop on Data Analysis, Vienna 2010