Wikipedia talk:Research

RFC: Proposal -> Policy

This policy should be implemented for the following reasons:

1. To allow important research of Wikipedia and its users to take place.
2. To ensure that that individual community members' preferences on whether and how frequently they are invited to participate in research studies are respected.
3. To ensure that research studies are respectful of participating community members and community norms.

I've notified all of the active members of Wikipedia:WikiProject_Research via talk page postings. --EpochFail^(talk|work) 15:49, 2 March 2010 (UTC)

Note that this policy has a sister approvals group that mediates the recruitment processes mentioned in this proposed policy. —Preceding unsigned comment added by PiperNigrum (talk • contribs) 16:30, 2 March 2010 (UTC)

• Support Wikipedia will benefit with this way to facilitate and control research into those who edit and use it. This policy provides a framework for researchers to operate in and a mechanism for Wikipedians to stop researchers contacting them. Josh Parris 15:07, 3 March 2010 (UTC)
• Support: This policy will help researchers carry out their research in a way that benefits Wikipedia, while respecting the wishes of members of the community. The needs of Wikipedia are carefully protected, while offering a way for researchers to carry out their studies. --Riedl (talk) 16:48, 12 March 2010 (UTC)

Typo?

"This policy describes how researchers can recruit members of the via talk page postings..." Members of the what? Steven Walling 19:03, 2 March 2010 (UTC)

Thanks for catching that.--EpochFail^(talk|work) 23:39, 2 March 2010 (UTC)

Making the raw data also open?

In practice, it is easy to make the results accessible for no fee by submitting a technical report e.g. to Arxiv before sending the publication to a journal / conference. A more interesting requirement, that would have huge benefit to the research community, would be the one of making the raw data available to all for free. So for instance if you do a study on whether people prefer revision A or B of articles, for many A-B pairs, the raw data results would be available to all. In this way, the volunteer time that went into an experiment could be reused for subsequent investigations. Of course, the data would have to be anonymized; in general, IRBs would have to approve of the raw data disclosure along with the rest of the project. We could have a wording that says "Once a study is completed, and the results are made available in publications, in three (or six?) months the authors will make available the raw data that was gathered as part of the study, after anonymizing it so that the identity of the participants is not disclosed". Or something to that effect. Lucadealfaro (talk) 19:44, 2 March 2010 (UTC)

♦ In principle, I agree; however, I worry that acceptably anonymizing the data will be prohibitively difficult in some cases and have a chilling effect - in other words, I see a tradeoff between more research with no raw data requirement and less research with a raw data requirement. I'd support a proposal to encourage release of suitably anonymized raw data but not one to require it. --R27182818 (talk) 22:27, 2 March 2010 (UTC)

Scope seems too narrow in a couple of regards

♦ The proposal contains the following core sentence: "This is the only way researchers are allowed to initiate contact via talk page postings to a large group of Wikipedia community members." Two thoughts. First, shouldn't SRAG regulate solicitations via other methods - for example, can't one e-mail users by some link rather than posting on talk pages? What about contacting WP users using a list of e-mail addresses one has from some other source? Second, what's the definition of large? Am I exempt from SRAG if I only need to solicit 2 users via talk page? 5? 10? 20? IMO, "to a large group of Wikipedia community members" should simply be struck. --R27182818 (talk) 22:34, 2 March 2010 (UTC)

If we strike the "to a large group of Wikipedia community members", we are left with 'This is the only way researchers are allowed to initiate contact via talk page postings' and then I'm not allowed to post on anyone's talk page without SRAG approval. Although I agree that "large" is hard to define, I think it is important that researchers are allowed to interact with community members without going through SRAG except for extreme circumstances (eg. mass recruitment). --EpochFail^(talk|work) 22:47, 2 March 2010 (UTC)

♦ Well, what about striking that text and expanding what "initiate contact" means - initiate contact for the purposes of recruitment? Or, strike the text and add some exceptions? Maybe you can give some examples of when you think approval is not necessary. If nothing else, "large" needs to be defined. I feel that this is a situation where researchers have incentive to interpret this sentence more generously than SRAG would like. --R27182818 (talk) 16:23, 3 March 2010 (UTC)

<--outdent-- How about something like the following:

A researcher must obtain SRAG approval for any talk page postings where:

• The researcher has not had prior communication with the user(s) being messaged
• The posting is not related directly to work on Wikipedia content, policy or systems.
• The posting is not personal communication.
• The posting was not solicited by the user.
• The user is not a member of WikiProject, work group, etc. managed by the researcher.
• The user is not or has not recently become a subject of one of the researchers studies.

As rule of thumb, if a researcher wants to post an unsolicited message on the talk page of an individual with the intent of collecting data from them for use in a study. That posting will require approval from WP:SRAG.

As you can see, I've dropped "large" in favor of being explicit about the types of message being posted. --EpochFail^(talk|work) 19:18, 3 March 2010 (UTC)

♦ I like it. How about the following modification (this also addresses my first concern):

A researcher must obtain SRAG approval to use tools within Wikipedia (e.g., talk page postings) to contact Wikipedia users when all of the following apply:

• The communication would be the first by that researcher and the user (automated talk page postings by bot do not count as prior communication).
• The message is not personal communication.
• The message is not related directly to work on Wikipedia content, policy, or systems.
• The user did not solicit the message.
• The user is not a member of a WikiProject, work group, etc. managed by the researcher.
• The user is not a current or recent participant in one of the researchers' studies.

As a rule if thumb, if a researcher wants to contact an individual with the intent of collecting data for use in a study, and that contact is unsolicited, he or she will need approval from WP:SRAG. --R27182818 (talk) 16:06, 4 March 2010 (UTC)

Agreed. I've added this within the current structure of the article. --EpochFail^(talk|work) 22:44, 5 March 2010 (UTC)

Where is the problem?

After reading this, I have to say this is WP:CREEP.

1. Are there cases where researchers canvassing for subjects have caused problems?
2. If the researchers act as users, they are part of the community, and have to live with the same rules and risks as everybody else, so this policy would be redundant. If they don't act as users, this policy is pointless to begin with.

Regards, Paradoctor (talk) 07:42, 6 March 2010 (UTC)

♦ Yes, see Wikipedia_talk:WikiProject_Research#Examples_of_unmediated_interactions. There have been several instances when studies have been derailed because of misunderstandings between researchers and the community. --R27182818 (talk) 16:57, 6 March 2010 (UTC)

The problem this policy tries to address is that "it is important to respect the wishes of Wikipedia community members on whether and how often they are contacted in this way". I won't let a committee decide that for me. What you can and should do is have a setting added to user preferences, or some user templates, e. g. {{i'm a guinea pig}}/{{use animals not wikipedians}}/{{property of SRAG}}. Paradoctor (talk) 21:48, 6 March 2010 (UTC)

The committee does not decide that for you. The policy is that you will have the ability to control whether or how often you would like to be contacted to be asked to participate via either {{bots}}, or a SubjectRecruitmentBot-specific template. -- PiperNigrum ^(hail|scan) 22:21, 6 March 2010 (UTC)

"The committee does not decide that for you.": Hm? How does that mesh with "Subject Recruitment Approvals Group (SRAG), a public discussion group, to control who will be allowed to make those postings"? Paradoctor (talk) 22:45, 6 March 2010 (UTC)

SRAG is the body that mediates the process of approving/denying researcher's requests to make postings to recruit users, and controls the SubjectRecruitmentBot that actually does the posting of messages. The determination of who receives these messages takes into account those who use the templates to indicate that they do not wish to be recruited or would only like to be recruited X times within Y time. Note that this deals with recruitment – the asking of users to participate in a study. Users who are recruited are by no means required to actually participate in the study if they do not want to. -- PiperNigrum ^(hail|scan) 00:01, 7 March 2010 (UTC)

What do we need a committee for when users can indicate their preferences? Paradoctor (talk) 00:15, 7 March 2010 (UTC)

The reason for SRAG is to be the body that oversees use of the bot. While users can indicate their preferences for contact, the bot shouldn't just sit there and allow anyone to use it to send messages to users. As part of approving use of the bot, researchers indicate what study they are conducting, what IRB approvals they have, and the type and number of users they would like to ask for participation, and then allows for discussion. Prior experience shows that there is a need for the community to have a means of identifying that research is being conducted. -- PiperNigrum ^(hail|scan) 16:11, 7 March 2010 (UTC)

"bot shouldn't just sit there and allow anyone to use it": So you want a committe overseeing MiszaBot III, too? Besides, automated messaging isn't exactly a difficult task. Give me list of usernames, Notepad++, and one hour for every 500 messages. For larger tasks, writing a little automation script is a nobrainer.

Please don't get me wrong, documentation for researchers and the bot are good ideas that fully deserve support. It's just that you're insisting on a buzzsaw where a kitchen knife is entirely sufficient. You have not demonstrated why a committee is essential to assisting researchers in acquiring participants. Try to see it from this perspective: Even the current, imperfect state of affairs seems tolerable, at least. Paradoctor (talk) 17:11, 7 March 2010 (UTC)

I agree that the bot itself would be simple to implement. However, I think that if someone were to submit a bot that does that to WP:BAG for researchers to use, they would not look favorably on it without some controls to prevent it from being misused for example to, for example, post commercial messages onto everybody's talk page.

Setting aside SRAG's bot responsibilities, the other purpose of SRAG is to give community acceptance to a study that is recruiting for subjects. One of the things researchers encounter on posting a call for participants are a lot of messages along the lines of "who the !@#$% are you", "do you even have IRB approval? I can't believe they'd grant it", "your study design is flawed, since you don't have a control group", etc. messages. Having SRAG as a place to post and discuss studies prior to recruitment would give the researchers a place to point to and say "It's gone through community discussion here." Now, I'm not saying that it will make all of that go away, but it would greatly cut down the number of these kinds of things.

-- PiperNigrum ^(hail|scan) 19:53, 7 March 2010 (UTC)

"commercial messages": Come on, I already said that spammers don't rely on our bots. And to sort out UCMs, we don't need a committee, edit protection for the bot's commando page is sufficient.

"community acceptance" ... "the things researchers encounter": That's something the researchers can handle themselves, e. g. Village Pump (misc). I don't think an anthropologist will parachute into the middle of the village and start shooting. ;) Paradoctor (talk) 20:18, 7 March 2010 (UTC)

Afterthought: If such a bot is going to be created, why not go the whole 9 mm, and make it a general WP:CanvassBot? Work is the same, and who knows what good use it might be put to? (Delivering Signpost?) Paradoctor (talk) 20:23, 7 March 2010 (UTC)

I'm with Paradoctor on this. we already have rules such as wp:canvass and I don't see the need for a filter where one group claim exclusive rights to vet research projects. I've taken part in a couple of research projects on the pedia and I do see the need for a project that aims to document all such initiatives, collate the findings and can advise researchers and hopefully prevent reinvention of wheels. But no I don't want a bureaucracy claiming exclusive rights to approve projects on the pedia. ϢereSpielChequers 00:17, 7 March 2010 (UTC)

One problem this policy is designed to improve is that researchers often face powerful blowback from users who question whether doing research on Wikipedia is appropriate at all. Having a review process will be important in responding to such users. I do think it's important not to see this as "one group claiming exclusive rights"; if you have experience in research on Wikipedia you should volunteer to help evaluating proposals. As with all of Wikipedia, the community should be making and enforcing the rules. It would be very unhealthy if somehow one group were to take over the execution of this policy. That is why many different researchers are helping to define the policy, and will be helping in its execution. --Riedl (talk) 23:55, 10 March 2010 (UTC)

It should be apparent that the proposed policy does not give SRAG exclusive rights to vet and approve all research projects. The scope of SRAG has been narrowly focused to deal specifically with research that requires contacting a large number of users via talk page postings. Research that does not involve this point need not go to SRAG at all.

As a successfully recruited participant in research, you may not see the difficulty researchers have in requesting participation from users of the community. As indicated in my reply above to Paradoctor, prior experience has shown that something like this is needed. It was suggested that rather than approaching Wikimedia and having something handed down à la WP:OFFICE, it would be more acceptable and wiki-like if we work from the bottom up and incorporate this into Wikipedia policy.

Regarding your other suggestion, WikiProject Research has been created for researchers to come and discuss other issues related to research of Wikipedia. Perhaps you should suggest your idea there.

-- PiperNigrum ^(hail|scan) 16:11, 7 March 2010 (UTC)

What SRAG is good for, maybe, from a semi-outsider.

I've done WP research before, as the person who for better or worse is the current owner of SuggestBot (though it's in the process of being moved to a more active caretaker). Someone asked me to read this discussion and comment. So:

Is the problem mostly the idea of "controlling who can post"? If the policy talked about establishing a broadly community-recognized mechanism for proposing, coordinating, legitimizing, and supporting both the execution of these kinds of many-posting experiments and tools for individual Wikipedians to opt out of them, would that be a clearer and more acceptable statement?

From what I'm reading one of the big goals is to have an experimental framework both that gives a standard best-practices way of conducting these things, and another is to be able to, when someone says "who the #*(Q%#@ are you?", point to having gone though SRAG as both a way to tell people about the academic research activity on Wikipedia and to be reassuring that some representative sample of Wikipedians has reviewed the experiment and found it to be "okay to do".

Maybe I'm reading this wrong, but that's what I see right now. It doesn't seem like a bad tool to have. I do think that people who want to do things their own way should be able to, but they face the practical issues that people on both sides of the policy have pointed about above, around how people react to the invitation/solicitation.

I'm probably not going to have too much more to contribute here, but this is what I see the discussion (so far) as being about. -- 128.84.103.49 (talk) 20:56, 10 March 2010 (UTC)

I think you have described the purpose of WP:Research and WP:SRAG correctly. Wikipedia should have WP:Research (and, of course, SRAG) as policy because research of Wikipedia is important and should take place, but it is more important that the community is in control of how researchers are allowed operate. Currently, randomly recruiting users via talk page postings is not covered by any policy, and for that reason, confusion and conflict are common. The community currently has no control and researchers do not have a way to go about recruiting users without strong backlash. This proposed policy and SRAG would both offer community control over when researchers get to recruit and give researchers legitimacy when they try to recruit subjects. --EpochFail^(talk|work) 22:05, 10 March 2010 (UTC)

Indicating general support

I see no harm with this, and potential good. Maurreen (talk) 22:43, 10 March 2010 (UTC)