User:Colin/A large scale student assignment – what could possibly go wrong?

In Autumn 2011, Steve Joordens ran an experiment on Wikipedia. He asked a very large number of students to make some relatively minor edits to part of Wikipedia. Joordens hoped this would be an effective way of harnessing both Wikipedia and the student population to publish research findings in psychology. This would benefit Wikipedia as psychology is a particularly weak topic, and would benefit the field of psychology as their research findings often had a limited academic audience. As professor of psychology at the University of Toronto at Scarborough, Joordens had access to over 1500 new students each term: those who took his “Introduction to Psychology” undergraduate course.

After the editing assignment completed, the experiment would assess the effect of the edits to Wikipedia articles, and the effect of the assignment on the students. Firstly, it would investigate whether the edits were retained, and the degree to which the edits were viewed by Wikipedia's readership. Secondly, it would observe whether the students continued to add information to Wikipedia voluntarily after the assignment.

The students earned bonus credits towards their grade as follows:

• 1% for creating a Wikipedia username and linking it to the Canadian Education Program, Registering with the APS Wikipedia Initiative, and linking their student number to the Wikipedia username
• 1% for editing any psychology-related article in the English language Wikipedia (an article either pulled from the list on the APS website, or one they found themselves) to include at least one sentence expanding on the article and including an appropriate citation to back up the claim made in the sentence.
• 1% for editing an additional article on a psychology-related topic either in the English version of Wikipedia or, preferably, in another language version they feel suitably proficient to edit.

The student assignment ran from 28 September 2011 to 1 December 2011. In the classroom, students were introduced to how Wikipedia would be used for the course and how it could affect their grade. They were provided handouts and links to guidelines and help pages.

Steve Joordens created his Wikipedia account – WoodSnake (talk · contribs) – on the 16th August 2011 and on the 26th September signed the Canada Education Program Memorandum of Understanding document. Other than creating his userpage, setting up the course pages on Wikipedia and commenting on the student leader-board, Joordens has made no edits to Wikipedia. In particular, he has not edited any articles on Wikipedia with the WoodSnake account. The assignment had four campus ambassadors to help the students. All of them were nearly as new to Wikipedia as the students. Two have never edited an article and the other two have made only minor edits to a few articles. None of them interacted on-Wiki with the other students or any Wikipedians during the assignment.

Independently of the experimentor's assessment, the student activities and their contributions were assessed by Wikipedians in detail at User:Colin/Introduction to Psychology, Part I. The following text is based on those notes.

Student analysis

Of the 1500+ students in the class, only 20% (317 students) performed the minimal step of creating a user account and registering on the assignment page. This would have taken around five minutes and was not a task the student could get wrong. Why was this no-brainer avoided by 80% of the students? Was the extra credit so insignificant that it wasn't worth a few minutes effort? If that is so, what does this say about the value the students would attach to the editing tasks?

Of those students who performed the first step, only half went on to edit Wikipedia articles: 10% of the total (158 students). Of these, a third gave up after editing just one article on English Wikipedia. It is possible that some of those chose to edit an article on another language version of Wikipedia but we can't track them. So only 7% of the total (107 students) completed the assignment on English Wikipedia by editing two or more articles.

Although the students had a couple of months in which to do the assignment, 85% of the edits were done after mid November and 28% were done after the 1st of December deadline. The students did not rush to complete this task early.

The 158 students edited 216 different articles. Although many of these were psychology-related, a large number were not and fell under the scope of biology and medicine, and in particular neuroscience. 54 articles were edited by more than one student. Typically, each article only received a few edits. The scope of the assignment was to just add one piece of information with a source.

Four of the 317 students who registered were already Wikipedians but only one of these participated further in the assignment by editing psychology articles. Students stuck closely to the assignment tasks by only editing their chosen articles and registering their usernames. They did not initiate or participate in article talk page or user talk page discussions. The students did not edit further subjects that were of personal interest and did not continue editing beyond the end of the assignment (though it is possible some students edit as an IP or with an alternative account).

Edit analysis

Analysing the contributions of the 317 registered students was a lengthy task. The student list was divided among three Wikipedians in the Medicine WikiProject (Colin, Doc James and Peter.C). None of them have training in psychology and they had limited access to the sources the students used so this imposed some constraints on the ability to detect bad edits or plagiarism. Due to the subjective nature of this analysis, and the different scoring methods used, it isn't possible to combine the results of the three groups of students.

Colin reviewed 116 students, of whom 63 edited articles. Of those 63, 11 (17%) added content that was a direct copy/paste of the source or too close a copy and so was removed. Note that where the students used class textbooks or journal articles behind a paywall, we were not generally able to detect plagiarism. A further 36 (57%) made edits that were so bad they were reverted. For example, totally inappropriate text, adding paragraphs of text to disambiguation-list articles, or writing so poor as to be incomprehensible. So of the 63, 47 (75%) made edits that left Wikipedia so much worse than before that their work was best removed. On the other hand, 31 (49%) made edits that added sourced information to Wikipedia that was acceptable.

Doc James reviewed 98 students, of who 49 edited articles. Of those 4 (8%) added copy/paste text and 1 (2%) added simple vandalism. 35 (71%) added unsourced content, or gave a source that wasn't appropriate, or added incomprehensible text. Just 9 (18%) added content that was ok.

Peter.C similarly found that for the most part students did not contribute in a way that was beneficial to Wikipedia's content and there were very few good edits made.

Students found it difficult to write proper citations. Since this is a requirement of any academic writing, it implies these students are not at an advanced stage. Frequent mistakes were merely writing "(Jones 2009)" but not supplying the full citation. Another common mistake was to forget the title of the journal article – perhaps the citation tool could make certain fields mandatory or at the least generate a warning should they be absent.

Content (which already existed on Wikipedia) was added to the wrong place. For example, adding prose to a disambiguation page. Some students didn't investigate the related articles (and summary-style hierarchy of articles) to work out which one deserved the information. Other students didn't carefully consider the scope of the article: for example, adding text on social isolation to the isolation (psychology) article, which discusses a psychological concept that is quite unrelated.

These students didn't understand their subject well enough to teach others (which is what encyclopaedic writing aims for). They made subject-level mistakes that even a non-psychologist could spot. They were utterly reliant on their chosen single source for material. This makes it very difficult for them to paraphrase their source material without making mistakes or just writing nonsense. Frequently, the text was incomprehensible (one could sometimes tell what they were trying to say if you already knew the subject or had read the source, but that's not the point).

Discussion

Psychology articles on Wikipedia are lousy. Many are stubs and not all are clearly scoped. Referencing is poor and often only general references are used if at all. While this may appear to make them an attractive target for an improvement exercise, they make very poor learning material and are difficult to work with.

Wikipedia has an unfortunate tendency towards detailing research studies when it should rather be merely stating the encyclopaedic facts. There is a place, of course, for articles and sections describing scientific research. However, students who are learning the science of psychology, and who are often using their textbooks or research papers as sources, will naturally fall into the trap of writing inappropriate content. The writing style of such sources (where facts are frequently attributed to the discovering researchers: Jones (2009) found that ....) is also inappropriate for Wikipedia.

Writing good encyclopaedic articles is hard. Geometry guy recently noted "Few people have the ability to do the necessary research, make the editorial judgements needed to select a good balance of source material, then compile this into a coherent narrative that will be a definitive treatment of the topic". A good article isn't just the accumulation of factoids by random editors. Three quarters of the students made edits that were so bad they were reverted. Those students who made acceptable edits did not lift the quality level of these already weak articles. They merely added more content. We can't expect these students to structurally improve the articles because that wasn't their assignment. Some of these articles are so bad they need a scythe taken to them.

Successful article writing depends on editor ability, motivation and support. Students studying the elementary aspects of their subject are no more able to write or contribute to a professional encyclopaedia article than any reasonably bright person. We really need to move up to the level of graduate student or those with academic or professional careers in the subject before their degree has much influence on the quality of their writing. Traditionally, Wikipedia contributors have been intrinsically motivated volunteers. Such volunteers can overcome their lack of education or experience through their enthusiasm for the subject. Student assignments mark a shift in the editor population towards extrinsically motivated writers. There is a pressure to complete the assignment even if one is not proud of the result – whereas a volunteer might give up or try something less ambitious. And not all the students will be any good. Previously, a dire essay would end up as so much paper in a drawer. Now such bad writing might harm an article read by thousands of people every day. Obviously, the opposite is also true: great students will get the chance to have their fine writing reach a worldwide audience. But these students aren't getting sufficient support.

The experiment assumed that bad material would not survive Wikipedia's quality control, and that edit retention could be used to semi-automate the assessment of students. This is a flawed assumption. Psychology is a neglected subject. And even on popular subjects, the actual number of committed Wikipedians able to police edits is generally over-estimated. That the plagiarism and poor content was reverted or fixed is almost wholly down to the extraordinary efforts of three Wikipedians. Several articles were subject to so many misguided student edits (not all from this assignment it has to be said) that they were wreaked had simply had to be reverted a good month or two back in time, with the odd worthy edit restored by hand.

These students were awarded a bonus credit for merely carrying out the activity. The actual content or source wasn't directly assessed. After complaints from some Wikipedians about plagiarism, students were reminded that this would count against them. Of course, given the potentially huge number of student edits (1500 students), it would be impossible for one professor and a few helpers to review the work in detail. And given that none of them were Wikipedians, the assessment couldn't include such aspects as medical sourcing guidelines or the manual of style. Since the extent of the new material added was typically a sentence or two, online plagiarism detection tools are ineffective.

These students did not become Wikipedians. They didn't edit other articles and haven't edited since. They didn't engage with the community or each other (on-wiki) at all.

Conclusion and recommendations

This assignment is due to be repeated beginning the 28th January and ending 9th April. One change planned by the institution is the additional credit warning:

• 1% will be deducted if text is added without backing it up with a reliable source OR if the reliable source is not properly cited. Whatever is added must be reworded from the original source. Copy and pasting information into Wikipedia is considered plagiarism and will be treated as a serious academic offence.

I don't believe that this change is sufficient to address the problems inherent in the assignment. Indeed, one would assume efforts are being made to improve on the 10% uptake for article edits. Therefore there is every reason to believe that the effect on Wikipedia will be worse. If 75% of the students make really bad edits (edits that should really be reverted) but nobody is willing to invest the considerable time involved in fixing those edits, then the psychology articles on Wikipedia will be damaged by this new assignment rather than improved. And even if all the bad edits were fixed, the net improvement is so slight that frankly, an experienced Wikipedian could contribute more and higher quality text in an afternoon given a decent source or two.

The January assignment should not go ahead as planned. If we are to expect 1500+ students to contribute to Wikipedia with little motivation and virtually no supervision, then we need to design an assignment that is far more likely to give a net benefit to Wikipedia. These students were ignorant of their subject and ignorant of Wikipedia and have been asked to perform a task that is beyond most of them to do well.

The WMF need to redesign their education material to emphasise the importance of professors and other tutors and helpers being active editors on Wikipedia. Doing the “Introduction to Wikipedia” tutorial shouldn't be merely a “consideration”. One can currently become a "Wikipedia Teaching Fellow" having only learned the basics of editing. This is just back-to-front thinking. Professors should take an article in their subject to GA level in order to learn the ropes. And any professor expecting their students to write GA level material should have written an FA. After all, we expect our professors to be at the very least the next level up in terms of experience and understanding of the subject they are teaching. Wikipedia should be no different.

Assignments should be planned in order to ensure that students have the necessary motivation to perform them well, and have the necessary knowledge of their subject matter and Wikipedia in order to write good encyclopaedic material. Not all assignments need involve adding content. The assessment of these assignments should include not only the usual academic review of the exposition of subject matter but also adherence to Wikipedia guidelines and policy. Those running the assignment should have sufficient helpers in place that they can deal with both editing and user issues in-house rather than rely on the Wikipedia community.

Students taking an elementary course in a subject will need much more supervision than those at graduate level. For many university subjects, students will not have a strong ability to write quality English prose, or to locate, cite and paraphrase their sources. We need to accept that not all students will be able to or be motivated to do a good job – it is up to the education establishment to remedy their student's mistakes and not to leave the task to overstretched volunteer Wikipedians.