Wikipedia:Community health initiative on English Wikipedia/Research about Administrators' Noticeboard Incidents
| Research about Administrators' Noticeboard Incidents |
|---|
|
|
|
Community health initiative on English Wikipedia |
Introduction[edit]
In 2017, Wikimedia Foundation Support and Safety team (SuSa) and the Anti-Harassment Tools team surveyed English-language Wikipedia contributors about their perception of Administrators' noticeboard/Incidents (AN/I), a noticeboard that is a key part of the conflict resolution process on the English Wikipedia. This survey is intended to understand community sentiments around AN/I, and will not lead to immediate or imposed changes to AN/I from the Foundation. Rather the purpose of the survey is to fill in gaps in data that could lead to on wiki discussions about possible improvements to how AN/I cases are managed. Any changes would need to be backed by the volunteer community on the English Wikipedia.
Methodology[edit]
The Wikimedia Foundation Support and Safety team and the Anti-Harassment Tools team recruited English Wikipedia contributors to participate in a survey about AN/I in order to identify possible areas of improvement for handling cases by learning about user experiences and satisfactions levels.
Invitations to take the survey were made on English Wikipedia noticeboards and invitations were placed on the user talk pages of 400 randomly selected active editors with recent edits to AN/I (defined as taking place within the previous 12 months, October 2016–October 2017). To involve the participation of a more diverse group of contributors, invitations to take the survey were sent to mailing lists of Wikimedia affiliates and some other known organized groups of English Wikipedia editors. The survey opened on November 28, 2017, and closed on December 8, 2017, at 23:00 UTC.
The survey asked the participants to complete 23 questions and included six with write-in answers. Answers from the six write-in questions were sorted and grouped, and then labeled to organize the feedback into cohesive ideas. Some answers received multiple labels as they provided multiple kinds of insight. The survey was broken into three parts:
- use of AN/I;
- satisfaction with AN/I; and
- experiences with AN/I.
The goal was to gauge what AN/I does well, what it doesn't do well, and what needs to be improved.
A total of 136 people filled out the survey. It is important to note that the following statistical analysis draws upon a survey with a small sample size of self selected experienced editors. As the survey was a voluntary opt-in survey, the sample of people who opted to respond to it might not be representative of the general Wikimedia user base. Furthermore, the findings presented in this report are based on the respondents' own assessment and understanding of their direct or secondhand experience with AN/I and, as such, they are limited in capturing the experiences only as perceived by the respondents. Other methods need to be explored in order to validate the information. We have gathered more information on the AN/I process in our quantitative data analysis; this data supplements the results of the survey from a more metrics-based perspective.
Demographics of participants[edit]
| How long have you been a Wikimedia contributor? | ||
| ||
0% did not select an option [?]
| ||
|
| Which gender do you identify with? | ||
| ||
0% did not select an option [?]
| ||
|
| Which region do you usually edit from? | ||
| ||
0% did not select an option [?]
| ||
|
Participation on AN/I[edit]
| How often have you reported incidents to AN/I in the last 12 months? | ||
| ||
0% did not select an option [?]
| ||
|
| How often have you been an uninvolved participant in discussions on AN/I in the last 12 months? | ||
| ||
0.74% did not select an option [?]
| ||
|
| How often have you been involved in an incident reported on AN/I in the last 12 months? | ||
| ||
0% did not select an option [?]
| ||
|
Key Findings[edit]
Theme 1: Satisfaction/dissatisfaction with AN/I[edit]
There are mixed opinions among respondents about how well the general process works to resolve AN/I cases. There is a stronger dissatisfaction expressed about the way that AN/I cases are handled rather than a negative view of the AN/I type process.
| On a scale of 1–5, how satisfied are you with the way reports are handled on AN/I? | ||
| ||
1.47% did not select an option [?]
| ||
|
| Have you avoided reporting one or more incidents to AN/I in the last 12 months, because you did not think it would be handled appropriately there? | ||
| ||
2.21% did not select an option [?]
| ||
|
| > If you did not select "no" above, why did you think those incidents would not be handled appropriately? | ||||
| ||||
74 participants (54.41%) did not enter a response for this question
|
One user wrote, "There is the boomerang factor. Who knows what I have in my history that people will comb through, so I put up with harassment or bullying or vandalism because the general rule is that boomerangs are allowed since there is no scope rule with ANI reports".
Another user mentioned bias specifically, "It depends on who shows up; bias comes into play, e.g. people take sides based on whether they personally agree with the actions of people, not if people's actions follow policies and guidelines".
| Have you ever disagreed with an AN/I outcome, such as the wrong person being sanctioned? | ||
| ||
1.47% did not select an option [?]
| ||
|
| On a scale of 1–5, do you agree with the general process of how AN/I reports work? (e.g report, discussion, and then decision?) | ||
| ||
2.21% did not select an option [?]
| ||
|
Theme 2: The type of case matters[edit]
| For which types of problems does AN/I work well? | ||||
| ||||
38 participants (27.94%) did not enter a response for this question
|
When asked "for which type of problems does AN/I work well," there were 98 write in responses. Responses were focused on specific kinds of problems, which we sorted into four major groups: the complexity of the problems, technical issues, rules, or problems involving editor conduct.
AN/I seems to work best for more 'obvious' or less complex problems like sock puppetry, "short term" problems (problems that can quickly or easily be resolved), copyright violations, and 'bot or automated script' problems. This seems to be reflected in the write-in answers, as well.
One user wrote, "AN/I works great for clear-cut rule violations, bot problems, etc. It does black-and-white very well."
Another described the issues AN/I handles well as "housekeeping and basic admin stuff, like IP vandalism/sockpuppetry, blatant trolling, etc."
Generally, the descriptions and suggestions were that AN/I handled cases best that were "objective and obvious cases" and "clear cut ones." Specifically, vandalism, bots, and sockpuppeting were mentioned frequently. In their responses, survey participants did not always include specific problems that AN/I is able to handle well; rather, they spoke about more general concepts. For example, lots of feedback mentioned 'straightforward cases' without specifying what makes a case straightforward.
Examples of answers that were labeled as 'straightforward cases': A user wrote in,"obvious instances of people breaking policies and guidelines."
A user wrote in, "relatively simple issues that just need one admin or don't really involve disputes; disputes where one editor is clearly the disruptive one and can be sanctioned."
| For which types of problems does AN/I NOT work well? | ||||
| ||||
37 participants (27.21%) did not enter a response for this question
|
This question had 99 write-in responses. Some of the responses have been grouped into multiple buckets since the answers fit multiple kinds of labels. The more complex, nuanced, complicated, or personal the issues, the more participants seemed to stress AN/I is not the right space to handle those problems. A complex problem may be something where the rules are ambiguous or contradictory, making solving a problem difficult, whereas a complicated problem could be a long running dispute between editors. Examples of these problems were described as cases that involved issues such as controversial topics, or issues involving civility problems and personal threats.
One user wrote in, "What AN/I doesn't do well is grey areas, long-term abuse, and taking action against "competence is required" editors whose problems are less clear cut."
Another user highlights this 'grey area' problem by saying, "Ambiguous issues, issues where the topic is ideologically divisive, or where one of the users is new and the other one is a 'old hat' with plenty of friends (rarely is the discussion unbiased in these cases)."
The more ambiguous and/or more investigative work required, the harder it is to solve. Longer term problems that require more back and forth were viewed by admins as harder to solve, and something that AN/I didn't solve well. One user suggests that longer term problems would be better solved on another board or space such as "AN or ArbCom."
A user wrote in, "Anything where looking at large amounts of evidence is needed, where the outcome isn't obvious or where one or more editors are long-term so have supporters and opposers."
Another user wrote, "Complex interpersonal disputes, content disputes that have been ongoing for long enough or involve enough editors to have turned into conduct problems, complaints related to "incivility", any complaint whose specific circumstances touch on broader wiki-political issues and which therefore encourages a lot of soapboxing and off-topic personal commentary from observers."
Ideas for improvements to AN/I[edit]
| If you could change one thing about AN/I, what would it be? | ||||
| ||||
26 participants (19.12%) did not enter a response for this question
|
| What are some other changes you would like to see in the AN/I reporting process? | ||||
| ||||
62 participants (45.59%) did not enter a response for this question
|
Theme 3: More moderators/clerks for AN/I discussions[edit]
A reoccurring theme in the free text comments was for there to be more moderators or clerks to enforce the rules. One person said, "discussions need to be clerked to keep them from raising more problems than they solve." Another user said, "clerking is needed to keep discussions focused."
| How often, in general, would you say the following statements apply to discussion on AN/I? — Discussion is focused and neutral. | ||
| ||
2.94% did not select an option [?]
| ||
|
Other responses indicate a general need for more administrators to monitor AN/I. A user suggested for there to be "more of the admin corpus active on any given day. It's the same old faces time after time."
One survey participant said that "too often I've seen meticulously documented reports being dismissed as TL;DR or just not acted upon because apparently no one wants to take the time to read the diffs and make a judgment call. AN/I also has its regulars who seem to delight in the drama (and often foment it), and the mischief they create sometimes complicates a thread so much that the core issues are lost. I also am a little tired of seeing complainants (myself included) criticized for choosing to file an AN/I report rather than pursing some other avenue. So, sometimes even if the report has its intended effect, the process of getting there is fraught with unpleasantness. For volunteer editors, this may not be worth it."
Another person said "an ideal solution would address the thanklessness of both the Admin's and the involved editor's role. Find a way to motivate Admins to spend more time, not less, per case. Redistribute the workload load and review Admin decisions for signs that they are overworked and becoming burnt out." Other suggestions for increasing participation include, have a rotating "watch officer" to review inputs and disposition some of them."
Closing of AN/I cases The survey indicates dissatisfaction with the way that AN/I cases are currently closed.
Theme 4: Policy[edit]
The survey finds that there are concerns around fairness of AN/I discussions that might be addressed with new policy. In particular, there were suggestions to make the discussions on AN/I fairer for new users. One respondent wants, "a greater commitment to fairness. Newcomers may not know all the ins and outs of Wikipedia but that doesn't automatically make them wrong in a dispute." Another comment suggests that the rule "Wikipedia:INVOLVED should be a rule for everybody at AN/I, not just admins. Otherwise 'popular' users often avoid heavy sanctions for issues that would get new editors banned."
There are mixed opinions among survey participants about the use of the BOOMERANG essay in AN/I discussions. As one user points out, "There is a tendency to issue "boomerangs" against the filer of a complaint -- mostly justified, but sometimes not so. Because of this, I try to make certain that the report is fully justified and of a serious-enough nature that it's worth the risk of an unjustified sanction being levied against me. I'd say in general, the AN/I community is overly fond of finding reasons for issuing boomerangs."
Additionally, the survey results show that some respondents want improved policies or enforcement of existing policy in order to positively change the tone and substance of the discussions on the AN/I during cases.
Other commenters want "restrictions on non-administrators participating in discussions, limiting them to a certain number of edits per month. They say that "this might attract admins to participate there more consistently."
Theme 5: More structure for reports to AN/I[edit]
While there was not concurrence about a specific type of change, there was strong feedback suggesting the benefit of modifying the structure of reports to AN/I.
| In your opinion, what changes should be made to how reports are made on AN/I?
Participants could select more than one answer | ||
| ||
5.88% did not select an option [?]
| ||
|
One participant points out, "How ANI is currently built is extremely conducive for agenda driven editors to watch the page all the time and continue drama and other "power maneuvers" against "other factions" I think having some fundamental changes in design could incentives people to be civil and/or look for peaceful resolution, rather than always try to escalate."
One respondent's suggestion is to "create an optional submission form that would (1) automatically generate links to mentioned users' contributions, (2) automatically notify those users, and (3) eliminate edit conflicts when initially filing a report." Other users wrote in suggestions like "Standardize reporting, to avoid walls of text", "Structured reporting (with opt-out allowed)" and "Use subpages for issues to avoid edit conflicts."
Theme 6: Other technical improvements[edit]
While there is not agreement about specific ideas, there is feedback that technical solutions could improve the function of AN/I. Some suggestions include:
- Tool for sorting/filtering posts
- Respondents also highlighted issues with uninvolved participants on the noticeboard; some contributions from participants uninvolved with the cases were seen as unhelpful or distracting. Some free text survey comments suggest to approach this problem by developing methods to sort or filter comments by participant.
- Private reporting system
- Some respondents want to provide an opportunity for anonymous, or privacy-shielded reporting that protects personal information about gender and other personal information that might be used for retaliation.
- Improved search feature
- The ability to search current and old cases to locate related cases.
- Twinkle or other gadgets to make reports and manage cases.
- New or improved gadgets for use on Wikimedia projects.
- Tool(s) to assist with finding and adding evidence to reports
- Better tools to search for and add diffs to reports.
Conclusions[edit]
Several comments by survey participants concisely summarize opinions about AN/I that were commonly stated in the survey.
| “ | I like its open nature: everyone with relevant information has the opportunity to provide it. Also, when it works, it works well. And sometimes it works perfectly: a succinct, informative report is made, an admin reads it, investigates, and takes action. That's relatively rare, however... | ” |
| “ | The instructions, form and process is too unstructured to be effective, reporters need the free time to keep responding, experienced users know how to manipulate the process in their favor, and administrators do not seem to have the time to be involved to investigate. | ” |
The results of this survey, similar to some community discussions around improvements to AN/I, do not point to a single point of improvement. However, the findings do strongly point to several areas where contributors see opportunities for improvements. Some would involve technical solutions, such as structural changes like form-based submission. Some would need social and procedural changes, such as the suggestions around clerking and moderation.
This research was prepared as part of a larger project to give Wikimedia communities more data when making decisions about their policies, and the processes used to enforce them. Support and Safety and the Anti-Harassment Tools team are committed to providing resources for community-supported change, be that technical development, further research, or coordination support.
We welcome discussion on the results on the talk page, and will be following and participating as they develop. If you have ideas about how to further research processes like AN/I, we would be happy to discuss them.