Function overview: Invites new good faith editors to play The Wikipedia Adventure. The Wikipedia Adventure is an interactive onboarding game designed to teach new contributors how to edit in a playful and educational fashion. It's funded by a Wikimedia Foundation Individual Engagement Grant and created by User:Ocaasi. The game takes place entirely in the editor's userspace and no outside Wikipedia pages are touched.
Links to relevant discussions (where appropriate):User:Ocaasi/TWAbag has the research and impact analysis plan and details
Edit period(s): Daily
Estimated number of pages affected: 100 per day during the course of the beta test (2-4 weeks)
We will invite a sample of 100 new editors to play TWA every day. The sample will be drawn from the set of users classified as “good faith” by the Snuggle tool developed by EpochFail. A sample of Snuggle data is available here. The criteria for invitation will be:
The user created their account within the past 24 hours
The user has made at least 1 main namespace edit
The user has a Snuggle desirability score of >.8. Blocked or banned accounts are excluded by this threshold, as are users who are likely to be editing in bad faith.
The user has not yet received a Teahouse invitation.
Invites to play the game will be sent via a talkpage invitation from HostBot. Users who receive an invitation and subsequently complete at least 1 level of the Wikipedia Adventure will serve as the Experimental group (Group A).
For every 100 editors invited to play TWA, another 100 new editors who meet the criteria for invitation will not receive one. Of these editors, those who subsequently make at least 1 edit to Wikipedia will serve as a basic experimental control group (Group B). We require at least 1 subsequent edit (after the hour when the user would have been invited to TWA, had they been included in Group A) in order to assure that the editors in this group would have had the opportunity to see the invitation--i.e. to make sure they had not already given up or lost interest in editing by the time of invitation.
A second control group (Group C) will consist of editors who received an invitation, did not play TWA at all, but who did make at least 1 edit to Wikipedia after receiving the invitation. This control group will be used to determine whether the invitation itself has any effect on subsequent editing activities, or long-term retention, separate from the potential impact of playing TWA.
The editing subsequent editing activities of the editors in Group A will be compared with those in Groups B and C. Metrics used to evaluate impact are likely to include number of edits, number of articles edited, change in Snuggle desirability score, and level of activity over time (retention), and may include other metrics.
Editors who start the game will be monitored for signs of increased vandalism to the encyclopedia, and cleanup actions will be taken by those monitoring as needed during the test.
While you're working on it, you should read the PEP8. →Σσς. (Sigma) 21:23, 9 November 2013 (UTC)
So, Wikipedia process and user experience improvement. Has this been discussed anywhere, or is it something off your own bat? Josh Parris 07:04, 8 November 2013 (UTC)
Ocaasi first proposed the Wikipedia Adventure over a year ago, and received support from the Wikimedia Foundation to move forward with the project earlier this year through an Individual Engagement Grant. The idea of a Wikipedia Adventure (and an initial prototype) was first conceived by Dcoetzee, I believe. There has been extensive discussion of the project over the past year, most prominently at The Wikipedia Adventure talk page. So, yes, it's very much Ocaasi's own bat :)
This bot request is for evaluation of the Wikipedia Adventure: which I am assisting with because I run a spambot, and I've done this kind of thing before.
Hi Josh Parris! Hopefully I can address some of your concerns:
Extensive alpha: As you can see, the alpha bugfixing was very successful. We patched 200 bugs with over 40 editors participating. There are under ten bugs left, all non-critical. We do not intend to run the beta test until these are fixed, so not ASAP, but soon. Since these are minor bugs, they shouldn't take more than a week. Even with a beta, bug-free software is always a goal but not a current condition and we'll continue actively patching bugs whenever they come up. You note that TWA was well supported on the talk page and globally as a grant proposal.
Editing userpages: As for editing new users' talk pages, there was an extensive discussion of this on AN/I and the consensus appeared to be that it was not a problem provided the editors were notified and given an upfront disclosure of the game's behavior (Wikipedia:Administrators'_noticeboard/IncidentArchive806#Message_forgery). Posting to editor's talk page is indeed a feature not a bug, as it permits the game to have a degree of realism and interactivity while remaining entirely separate from the project's public-facing article pages. Several concrete steps were taken towards the end of disclosure: 1) All automatic posts have identifying edit summaries that they were "simulated automatically as part of The Wikipedia Adventure"; 2) There is an infobox on WP:TWA that states automatic messages are sent to userspace; 3) The second slide of the adventure reiterates that point that there are automatic messages sent.
Experimental focus: The purpose of this test is to gather data to evaluate the Adventure's effectiveness at increasing editor activity and retention and constructive behavior. We have no intent to expand beyond this initial test before we have data clearly demonstrating the test is unproblematic and then further that the game is effective. Indeed, after this moderately-sized beta test, we would take time to evaluate and analyze the data while also sharing it anew with the community to determine if further testing is needed and appropriate, or if a broader expansion would be useful and beneficial.
Do you have any other ideas that would help allay your concerns or further demonstrate consensus for a limited experimental beta-test? I'd be happy to take those steps if you have more ideas. Best, Ocaasit | c 09:07, 9 November 2013 (UTC)
Everything you say makes sense, and I support you trying this out. ANI seems to read as "You did a confusing thing! Oh, neat idea. Sounds good, I like it!" But again, there doesn't seem to have been any point where you've gone to the community and said "We're going to go and message 100 new editors, is that okay?". Messaging editors is something the community has historically been sensitive about. Go to WP:VPR, and if they don't say it's not okay, then you're in (I'm allowing for a tumbleweed response, which is not unprecedented). Josh Parris 11:03, 9 November 2013 (UTC)
┌────────────────────────────────────────────────────────────────────────────────────────────────────┘ Hi Josh Parris, it's been a few days and there's zero activity. Any thoughts on how to proceed? Cheers, Ocaasit | c 10:57, 13 November 2013 (UTC)
Will your A-B testing be screwed up if you don't do the hundred in one hit? If not, then Approved for trial (5 edits)., otherwise speak up. Josh Parris 11:00, 13 November 2013 (UTC)
Hi Josh Parris. We can test out the bot function with 5 edits, however, in order to have any statistically valid data by the trial's end were going to need to send the invite to several hundreds of users. Most users who receive the invite will not play the game, so in order to have valid data from those who do play the game, we need a significantly larger sample. Our goal is to have that data during November and early December, for extensive analysis by January. I'm fully on board to have us test out the code in a small batch; however, we are looking for a broader implementation in order to gauge The Wikipedia Adventure's impact across a broad sample of new editors. Please let me know if I can explain our test plan any further. Thanks! Ocaasit | c 23:16, 14 November 2013 (UTC)
I fully understand your task. Let's just treat the five edits as a sanity test. Off you go! Josh Parris 04:16, 15 November 2013 (UTC)
Sweet! We're running the test today, Josh Parris. Thanks for stepping up and facilitating this process :) Ocaasit | c 11:09, 15 November 2013 (UTC)