Observer bias

Background

Observer bias is one of the types of detection bias and is defined as any kind of systematic divergence from accurate facts during observation and the recording of data and information in studies. The definition can be further expanded upon to include the systematic difference between what is observed due to variation in observers, and what the true value is.

Observer bias is the tendency of observers to not see what is there, but instead to see what they expect or want to see. This is a common occurrence in the everyday lives of many and is a significant problem that is sometimes encountered in scientific research and studies. Observation is critical to scientific research and activity, and as such, observer bias may be as well. When such biases exist, scientific studies can result in an over- or underestimation of what is true and accurate, which compromises the validity of the findings and results of the study, even if all other designs and procedures in the study were appropriate.

Observational data forms the foundation of a significant body of knowledge. Observation is a method of data collection and falls into the category of qualitative research techniques. There are a number of benefits of observation, including its simplicity as a data collection method and its usefulness for hypotheses. Simultaneously, there are many limitations and disadvantages in the observation process, including the potential lack of reliability, poor validity, and faulty perception. Participants’ observations are widely used in sociological and anthropological studies, while systematic observation is used where researchers need to collect data without participants direct interactions. The most common observation method is naturalistic observation, where subjects are observed in their natural environments with the goal to assess the behaviour in an intervention free and natural setting.

Observer bias is especially probable when the investigator or researcher has vested interests in the outcome of the research or has strong preconceptions. Coupled with ambiguous underlying data and a subjective scoring method, these three factors contribute heavily to the incidence of observer bias.

Examples of cognitive biases can be seen in the table below.

Bias

Anchoring – a cognitive bias that causes humans to place too much reliance on the initial pieces of information they are provided with for a topic. This causes a skew in judgement and prevents humans and observers from updating their plans and predictions as appropriate.

Bandwagon effect – the tendency for people to “jump on the bandwagon” with certain behaviours and attitudes, meaning that they adopt particular ways of doings things based on what others are doing.

Bias blind spot – the tendency for people to recognize the impact of bias on others and their judgements, while simultaneously failing to acknowledge and recognize the impact that their own biases have on their own judgement.

Confirmation bias – the tendency for people to look for, interpret, and recall information in such a way that their preconceived beliefs and values are affirmed.

Guilt and innocence by association bias – the tendency for people to hold an assumption that individuals within a group share similar characteristics and behaviours, including those that would hail them as innocent or guilty.

Halo effect – the tendency for the positive impressions and beliefs in one area around a person, brand, company, product or the like to influence an observers opinions or feelings in other unrelated areas.

Framing effect – the tendency for people to form conclusions and opinions based on whether the pertinent relevant is provided to them with positive or negative connotations.

Recency effect – the tendency for more recent pieces of information, ideas, or arguments to be remembered more clearly than those that preceded.

Examples

Examples of observer bias extend back to the early 1900’s. One of the first recorded events of apparent observer bias was seen in 1904, with the case of “Clever Hans”. Clever Hans was a horse, whose owner, Wilhem Von Olson, claimed could perform arithmetic equations. Von Olson would ask Clever Hans a series of questions involving arithmetic functions, and the horse would appear to answer by tapping its hoof with the numbered answer. This example was investigated by the psychologist Oskar Pfungst, and it was found that when the horse was nearing the correct number of taps, the owner would subconsciously react in a particular way, which signalled to Clever Hans to discontinue his tapping. This only worked, however, when the owner himself knew the answer to the question. This is an example of observer bias, due to the fact that the expectations of Von Olson, the horses’ owner, were the cause of Clever Hans actions and behaviours, resulting in faulty data.

One of the most notorious examples of observer bias is seen in the studies and contributions of Cyril Burt, an English psychologist and geneticist who purported the heritability of IQ. Burt believed, and thus demonstrated through his research because of his observer bias, that children from families with lower socioeconomic status were likely to have lower levels of cognitive abilities when compared with that of children from families with higher socioeconomic status. Such research and findings had considerable impacts on the educational system in England throughout the 1960s, where middle- and upper-class children were sent to elite schools while the children from the lower socioeconomic demographic were sent to schools with less desirable traits. Following Burt’s death, further research found that the data in Burt’s studies was fabricated, which was presumed to be a result of his observer bias and the outcomes he was intending to find through his studies.

Impact of Observer Bias

Observational data forms the foundation of a significant body of knowledge. Observer bias can be seen as a significant issue in medical research and treatment. There is greater potential for variance in observations made where subjective judgement is required, when compared with observation of objective data where there is a much lower risk of observer bias.

When there is observer bias present in research and studies, the data collection itself is affected. The findings and results are not accurate representations of reality, due to the influence of the observers’ biases. Although they may not intend to do so, observer bias may result in researchers subconsciously encouraging certain results, which would lead to changes in the findings and outcomes in the study. A researcher that has not taken steps to mitigate observer bias and is being influenced by their own observer bias has a higher probability of making erroneous interpretations, which ultimately will lead to inaccurate results and findings.

Research has shown that in the presence of observer bias in outcome assessment, it is possible for treatment effect estimates to be exaggerated by between a third to two-thirds, symbolising significant implications on the validity of the findings and results of studies and procedures.

Preventative steps to mitigate observer bias

Bias in unfortunately an unavoidable problem in epidemiological and clinical research. However, there are a number of potential strategies and solutions for the reduction of observer bias, specifically in the areas of scientific studies and research across the medical field. The effects that bias has can be reduced through the use of strong operational definitions, along with masking, triangulation, and standardisation of procedures, and the continual monitoring of the objectivity of those conducting the experiments and observations.

Blinded protocols and double-blinded research can act as a corrective lens in terms of reducing observer bias, and thus, to increase the reliability and accuracy of the data collected. Blind trials are often required in order for the attainment of regulatory approval for medical devices and drugs, but are not common practice in empirical studies despite the research supporting its necessity. Double-blinding is done by ensuring both the tester and research participants lack of information that could have a potential influence on their behaviour, while single-blind describes those experiments where information is withheld from the participants that may otherwise skew the results or introduce bias, but the experimenter is entirely aware of and in possession of those facts.

An example of how observer bias can impact on research, and how blinded protocols can impact, can be seen in the trial for an anti-psychotic drug. Researchers that know which of the subjects received the placebo and those that received the trial drugs may later report that the group that received the trial drugs had a calmer disposition, due to the expectations of that outcome. Similarly, if the participants in the trial were not blinded, then they may report how they are feeling differently based on whether they were provided with the placebo or the trial drug.

A further example could be seen at schools. Boys of school-age generally outperform their female peers in science, however there is evidence that this is potentially as a result of how they are taught and treated by their teachers, who have the expectation that the boys have higher performances, and thus subtly encourage them. As such, the observers, being the teachers who conduct tests and evaluate the results, have a bias and preconceived belief that boys will outperform girls, which impacts on their behaviour.

To complement blind or masked protocols and research, further strategies including standardised training for observers and researchers about how to record findings can be useful in the mitigation of observer bias.  Clear definition of methodology, tools and the time frames allocated for the collection of findings can assist in adequately training and preparing observers in a standardised manner. Further, identifying any potential conflicts of interest within observers before commencement of the research is essential in ensuring bias is minimised.

Finally, triangulation within research is a method that can be used to increase the findings validity and credibility. Triangulation in research refers to the use of a variety of methods or data sources as a means of developing a more comprehensive and accurate understanding of the subject at hand. Triangulation will considerably increase the confidence in a study tremendously. There are a few ways triangulation can occur, including the use of multiple observers, which is a form of reliability in itself called interobserver reliability, measured by the percentage of times that the observers agree.

Hawthorne Effect (Observer Effect)

Observer bias is commonly only identified in the observers, however, there also exists a bias for those being studied. Named after a series of experiments conducted by Elton Mayo between 1924 and 1932, at the Western Electric factory in Hawthorne, Chicago, the Hawthorne effect symbolises where the participants in a study change their behaviour due to the fact that they are being observed.

Within the Hawthorne studies, it was found that the departmental outputs increased each time a change was made, even when the changes made were reverting to the original unfavourable conditions. The subjects in the experiment were told that better lighting would result in improved productivity, and as such, their beliefs about the impact of good lighting had a more significant effect on their behaviour and output than what the actual lighting levels were. Researchers formed the conclusion that the workers were in fact responding to the attention of the supervisors, not the changes in the experimental variables.

To prevent the Hawthorne Effect, studies using hidden observation can be useful, however knowledge of participation in the study would be required by law and is thought to still have the potential to cause the induction of the Hawthorne Effect. Further, making responses or study data completely anonymous will result in reducing the likelihood of participants altering their behaviour as a result of being observed as they take part in an experiment or study. Furthermore, conducting research prior to the studies to establish a baseline measure could assist in mitigating the Hawthorne Effect from biasing the studies results significantly. With a baseline established, any potential participant bias that arises as a result of being observed can be evaluated. Furthermore, establishing a follow up period could be of benefit to enable the examination of whether a behaviour or change continues and is sustained beyond the observation period.

See also

• Observer-expectancy effect, when a researcher subconsciously influences the participants of an experiment