Situational judgement test

From Wikipedia, the free encyclopedia
Jump to: navigation, search

Situational judgment tests (SJTs) or Inventories (SJIs) are a type of psychological test which present the test-taker with realistic, hypothetical scenarios and ask the individual to identify the most appropriate response or to rank the responses in the order they feel is most effective.[1] SJTs can be presented to test-takers through a variety of modalities, such as booklets, films, or audio recordings.[2] SJTs represent a distinct psychometric approach from the common knowledge-based multiple choice item.[1][3] They are often used in industrial-organizational psychology applications such as personnel selection. Situational judgment tests tend to determine behavioral tendencies, assessing how an individual will behave in a certain situation, and knowledge instruction, which evaluates the effectiveness of possible responses.[4] Situational judgment tests could also reinforce the status quo with an organization.[4]

Unlike most psychological tests SJTs are not acquired 'off-the-shelf', but are in fact designed as a bespoke tool, tailor-made to suit the individual role requirements.[1] This is because SJTs are not a type of test with respect to their content, but are a method of designing tests.

Developing a Situational Judgment Test[edit]

Developing a situational judgment test begins with conducting a job analysis that includes collecting critical incidents. These critical incidents are used to develop different situations where the judgment of the prospective new hire would need to make a decision. Once these situations are developed, subject matter experts (excellent employees) are asked to suggest effective and less effective solutions to the situation. Then a different group of subject matter experts rate these responses from best to worst and the test is scored with the highest ranked options giving the respondent the higher score (or lower if the test is reverse scored).[5]

Validity[edit]

The validity of the test corresponds to the types of questions are being asked. Knowledge instruction questions correlate more highly with general mental ability while behavioral tendency questions correlate more highly with personality.[4]

Key results from a study show that knowledge about interpersonal behavior measured with situational judgment tests was valid for internships (7 years later) as well as job performance (9 years later). Also, students' knowledge of interpersonal behavior showed progressive validity over cognitive factors for predicting academic and post academic success. This study was also the first study to show evidence of the predictive long-term power of interpersonal skill assessed though situational judgment tests.[6]

There are many problems within scoring SJTs. "Attempts to address this issue include expert-novice differences, where an item is scored in the direction favoring the experts after the average ratings of experts and novices on each item are compared; expert judgment, where a team of experts decides the best answer to each question; target scoring, where the test author determines the correct answer; and consensual scoring, where a score is allocated to each option according to the percentage of people choosing that option."[7]

History[edit]

The situational judgment test has been around for over fifty years. The first two that were documented were the How supervise and the Cardall Practical Judgment Test. In 1958 the Supervisory Practice Test came about by Bruce and Learner.[8] The Supervisory Practice Test was to point out whether or not supervisors could handle certain situations on the job. This test is said to effectively identify who could and could not be a supervisor.[8] The situational judgment test did not really take off and become a great use in the employment field until the early 1990s.[8]

Situational Judgment Tests then went on to be used in World War II by psychologists in the US military.[3] "In the 1950s and 60s, their use was extended to predict, as well as assess,managerial success."[9]

Today, SJTs are used in many organizations, are promoted by various consulting firms, and are researched by many.[3]

Tests to Measure Individual Adaptability in Applied Settings[edit]

A Thesis Submitted to George Mason University in 2010 by Adam M. Grim created a study to measure individual adaptability in applied settings. An Adaptability Situational Judgment Test (ASJT) was designed to provide a practical and valid selection and assessment instrument that had incremental validity beyond the Big Five personality traits and cognitive ability in predicting supervisor ratings of adaptability.[10] "The research contributes to the selection and adaptive performance literatures by demonstrating that it is possible to use a situational judgment test to measure individual adaptability in both military and non-military applied settings."[10] ASJT had similar relationships with all variables of interest in both samples, thus providing support for the generalizability of the measure to both military and business settings. Practical implications and recommendations for future refinements of ASJT are discussed.[10] With this ASJT did not have differential validity and provides a selection instrument that would not cause adverse impact or be subject to legal challenge because of predictive bias.[10] For this study there were both business and military setting scenarios which subjects would read and indicate how likely they were to do the list of behaviors related to that scenario.[10]

Multiple-choice Examples[edit]

Consist of either taking the test on paper or written out examples online. The online version offers a few advantages such as, faster results. It is often the case that Situational Judgment Test have multiple correct answers even though an answer might be more preferred by the hiring organization.[4]

You are the leader of a manufacturing team that works with heavy machinery. One of your productions operators tells you that one machine in the work area is suddenly malfunctioning and may endanger the welfare of your work team. Rank order the following possible courses of action to effectively address this problem. from most desirable to least desirable. 1. Call a meeting of your team members to discuss the problem. 2. Report the problem to the Director of Safety 3. shut off the machine immediately. 4. Individually ask other production operators about problems with their machines. 5.evacuate your team from the production facility.[4] Other typical examples can be found here.[11]

Video-based Examples[edit]

Consists of videos that contain different scenarios that the employee may face. Scenarios for this section can be found on youtube.com.

Scenarios are in many different styles such as:

  • Animated people and situations.
  • The boss of the company could be recorded asking the question.
  • The answering process can be different for each test.
* The correct answer could be given.
* The individual could be ask to give the most reasonable answer.
* The individual is asked to explain what they were to do if they were in that situation.

Advantages over other measures[edit]

  • They show reduced levels of adverse impact, by gender and ethnicity,[12] compared to cognitive ability tests.[1][13]
  • They use measures that directly assess job relevant behaviours.[1]
  • They can be administered in bulk, either via pen and paper or on-line.[1]
  • The SJT design process results in higher relevance of content than other psychometric assessments[13][14] They are therefore more acceptable and engaging to candidates compared to cognitive ability tests since scenarios are based on real incidents[1]
  • It is unlikely that practice will enhance candidate performance as the answers cannot be arrived at logically – a response to a situation may be appropriate in one organisation and inappropriate in another.[1]
  • They can tap into a variety of constructs – ranging from problem solving and decision making to interpersonal skills.[1] Traditional psychometric tests do not account for the interaction between ability, personality and other traits.[13]
  • Conscientiousness can be built into a test as a major factor of individual differences.[15]
  • They can be used in combination with a knowledge based test to give a better overall picture of a candidate's aptitude for a certain job.[16]

Company Use[edit]

Companies using SJTs report the following anecdotal evidence supporting the use of SJT. Note: these reports are not supported by peer reviewed research.

  • Can highlight employee developmental needs[17]
  • They are relatively easy and cost-effective to develop, administer and score[17]
  • There has been more favorable applicant reactions to this test than to general mental ability tests.

Criticisms[edit]

  • The scenarios in many SJTs tend to be brief; therefore candidates do not become fully immersed in the scenario. This can remove some of the intended realism of the scenario and may reduce the quality and depth of assessment.[13]
  • SJT responses can be transparent, providing more of an index of best practice knowledge in some cases and therefore failing to differentiate between candidates' work-related performance.[13]
  • The response formats in some SJTs do not present a full enough range of responses to the scenario. Candidates can be forced to select actions or responses that do not necessarily fit their behavior. They can find this frustrating and this can affect the validity of such measures[18][19][20]
  • Because of the adaptability of SJTs, arguments persist about whether or not they are a valid measurement of a particular construct (Job Knowledge), or a measurement tool which can be applied to a variety of different constructs, such as cognitive ability, conscientiousness, agreeableness, or emotional stability[21]
  • SJTs are best suited for assessing multiple constructs, and as such, it is difficult to separate the constructs assessed in the test. If one construct is of particular interest, a different measure may be more practical.[22]
  • Due to the multi-dimensional nature of SJTs, it is problematic to assess reliability through the use of standard measures.[23]

Notes[edit]

  1. ^ a b c d e f g h i "Situational Judgement Tests: Are they just measures of cognitive ability?". Human Assets. Retrieved August 5, 2011. 
  2. ^ Lievens, Filip; Helga Peeters; Eveline Schollaert (2008). "Situational judgment tests: a review of recent research". Personnel Review 37 (4): 426–441. doi:10.1108/00483480810877598. 
  3. ^ a b c McDaniel, Michael A.; Whetzel, Deborah L. "Situational Judgment Tests. An IPMAAC Workshop" (PDF). IPMA-HR Assessment Council. Retrieved August 7, 2007. 
  4. ^ a b c d e Muchinsky, Paul (2012). Psycholofy Applied to Work. Summerfield, NC: Hypergraphic Press. pp. 110–11. ISBN 978-0-578-07692-8. 
  5. ^ Lievens, Filip (2008). Personnel Review 37 (4): 426–441. doi:10.1108/00483480810877598. 
  6. ^ Lievens, Filip; Paul R. Sackett (1 March 2012). "The Validity of Interpersonal Skills Assessment Via Situational Judgment Tests for Predicting Academic Success and Job Performance". Applied Psychology 97 (2): 460–468. doi:10.1037/a0025741. 
  7. ^ http://eprints.usq.edu.au/787/1/Strahan_Fogarty_Machin_APS_Conference_proceedings.pdf
  8. ^ a b c Swander, Carl (3 May 2001). "Video-based Situational Judgement Test Characteristics: Multidimensionality at the Item Level and Impact of Situational Variables". pp. 3–5. 
  9. ^ http://eprints.usq.edu.au/787/
  10. ^ a b c d e Grim, Adam. "Use of Situational Judgment Test to Measure Individual Adaptability in Applied Settings". Retrieved March 1, 2012. 
  11. ^ "Example Situational Judgement Test Items". 
  12. ^ Hoare, S., Day, A., & Smith, M. (1998). The development and evaluation of situations inventories. Selection & Development Review, 14(6), 3-8.
  13. ^ a b c d e "Technical Information". Harcourt Assessment. Retrieved August 7, 2007. 
  14. ^ Motowildo, S.J., Hanson, M.A., & Crafts, J.L. (1997). Low fidelity simulations. In D.L. Whetzel & G.R. Wheaton (Eds.), Applied Measurement in industrial Psychology. Palo Alto, CA: Davies-Black.
  15. ^ McDaniel, Michael. & Nguyen, Nhung "Situational Judgement Tests: A Review of Practice and Constructs Assessed", Blackwell Publishers LTD, Oxford, March/June 2001. Retrieved on 17 October 2012.
  16. ^ Rahman, Mahibur. "Tackling situational judgment tests". BMJ Publishing Group Ltd. Retrieved August 7, 2007. 
  17. ^ a b "What Is The SJT". Medical Schools Council. Retrieved October 16, 2012. 
  18. ^ Chan, D., & Schmitt, N. (2005). An agenda for future research on applicants' reactions to selection procedures: A construct-orientated approach. International Journal of Selection and Assessment, 12, 9-23.
  19. ^ Ployhart, R.E., & Harold, C.M. (2004). The applicant attribution-reaction theory (AART): An integrative approach of applicant attributional processing. International Journal of Selection & Assessment, 12, 84-98.
  20. ^ Schmit, M.J., & Ryan, A.M. (1992). Test-taking dispositions: A missing link? Journal of Applied Psychology, 77, 629-637.
  21. ^ McDaniel, M.A., Morgeson, F.P., Finnegan, E.B., Campion, M.A., & Braverman, E.P. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740.001
  22. ^ McDaniel, M.A., Morgeson, F.P., Finnegan, E.B., Campion, M.A., & Braverman, E.P. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740.
  23. ^ McDaniel, M.A. & Whetzel, D.L. (2007). Situational Judgement Tests. In D.L. Whetzel & G.R. Wheaton (Eds). Applied measurement: Industrial psychology in human resources management. Erlbaum. 235-258.

See also[edit]