Wikipedia talk:Research/Archive 1

Page contents not supported in other languages.
From Wikipedia, the free encyclopedia
Archive 1 Archive 2

Statistical validity

You're going to need some way to ensure that survey respondents are statistically representative, if you do research on subjects just because they agree to allow you to there's immediately going to be systematic bias. I suggest some sort of drive to get people to volunteer to identify themselves as willing to respond to surveys by adding a userbox or category, and then select from that group to get an appropriately representative sample. Josh Parris 06:15, 2 December 2009 (UTC)

Although I agree with you that having editors opt-in to studies introduces bias into the results, this type of bias is relatively inescapable unless we can find a way to have editors take a survey/use an experimental interface whether they want to or not without violating their rights or privacy. I'm very skeptical that such an approach would be possible. On the other hand, I feel that the purpose of this policy is to protect editors from researchers--not to ensure that researchers get generalizable results. It seems to me that our primary concern with this policy should be that editors are not "bothered" by requests from researchers to participate in studies, but that researchers are allowed to sample from them. Please see a discussion of mechanisms for addressing this issue. --EpochFail(talk|work) 17:23, 2 December 2009 (UTC)
And we've seen a recent example of what can happen if you make people use something whether they want to or not, courtesy of the Usability Initiative. -- PiperNigrum (hail|scan) 18:11, 2 December 2009 (UTC)

Opt-out or Opt-in?

"Editors who do not wish to receive study recruitment messages will be able to opt out"

I was suggesting an opt-in process rather than an opt-in default because, well, it's annoying. I don't think one generic question along the lines of "Would you be prepared to answer questions to assist academic research into Wikipedia?" with a opt-in template slapped on the user-page if they answer "yes" would be a problem; subsequent questions of "Can researchers contact you by email?" and "Would you be prepared to answer questions to assist academic research into the sexual habits of Wikipedia users?" could then be filtered based on that broad participation question. But asking users "So, how often do you have sex with your partner, and is that affecting your editing on Wikipedia? If you don't want to partipate in academic surveys, just put {{this template}} on your userpage" I don't think cuts it. Josh Parris 23:22, 2 December 2009 (UTC)

Actually, in contrast to your second example, I would imagine a recruitment message would be something like the following:
You are invited to participate in an academic survey of the sexual habits of Wikipedia users. Please see [[this page]] for more information on this survey, or to participate. If you do not want to be invited to participate in academic studies, just put {{this template}} on your user page.
This is very similar to editor's interactions with other bots, such as the SineBot example, and seems well within the bounds of an acceptable means/level of communication. -- PiperNigrum (hail|scan) 02:26, 3 December 2009 (UTC)
I agree. The difficulty would be the first research project that comes along and actually annoys people will close the door on all subsequent projects; the SineBot template, on the other hand, just affects signing. Also, if someone generically agrees to academic studies, then they've taken the first step in an escalation of commitment so subsequent questions ("Well, if you're willing to participate in academic studies, you're willing to have a quick peek at [[this study on what effects your editing time]] and perhaps suggest volunteers - maybe yourself? - as study subjects") are more likely to succeed.
A shiny user-box "This user expands human knowledge by being a research guinea pig" would help. People seem to love user boxes.
So, first message would be of the "Would you be willing to be approached to answer questions to assist academic research into Wikipedia? If not, just put {{this template}} on your user or talk page." type, and after a week you'd come back to those who hadn't explicitly blocked you. In fact, initially, you could run it both ways (with both opt-in and opt-out templates) and see how this affects the quality and quantity of your results. Josh Parris 03:14, 3 December 2009 (UTC)
I'm confused about what you are trying to get at here. Do you mean to suggest that there should be a post made to a large group of user talk pages with a message asking them if they would be interested in participating in studies in general? It seems to me that we would have had already committed the faux pas we are trying to avoid--bothering users unnecessarily.
It is important to remember that the opt-out approach we are discussing is for recruitment alone. No one is suggesting that the actual studies themselves be opt-out (like the Usability Initiative work). We are merely talking about very "infrequent" postings to user talk pages asking if they would be interested in participating in particular studies to which a user could say, "Never bother me with this again." Looking through the archives of my own talk page, nearly half of the messages were posted by bots and gadgets. --EpochFail(talk|work) 15:55, 3 December 2009 (UTC)
Thing is, the existing bot messages come as a result from you doing things, which invites comment and communication. This bot would leave a message regardless of your activity. I'm not saying you ought not do it, it's just a departure from existing practice. However, if you make the selection bot offer participation as a function of recent editing, you will skew your sample even if you randomly offer based on that user's historical edit frequency (assuming an editor doing 10K edits per month ought to have the same selection probability as one doing 10/month).
I think I'm saying in communication with editors, there ought to be an explanation that they're being asked because they're an editor, and that they were randomly selected by a bot (possibly along with the criteria), and that this isn't a blanket approach to each and every editor, and no, we're not spamming your talk page. "It's business, not personal". Josh Parris 23:45, 3 December 2009 (UTC)
This is one of the goals of having this policy -- to inform editors that there are people who are interested in conducting research on Wikipedia and that they may be randomly selected to participate in a study by a bot, and that this is being managed and not spamming. -- PiperNigrum (hail|scan) 02:52, 4 December 2009 (UTC)

Changes to accepted studies

If the study change does not affect the sampling requirements and the recruitment messages have already been posted, no interaction with WP:SRAG is necessary.

So, if I said I was going to do ink-blot testing but changed to reactions to the depiction of animal cruelty, that would be fine? Josh Parris 00:11, 4 December 2009 (UTC)

A situation like this should never reach the SRAG as a study change. If you wanted to change your ink-blot testing to reactions of the depictions of animal cruelty, you'd have to run that change by your institution's IRB first, and I have serious doubts that they would accept that as a change to an existing study rather than a new study. -- PiperNigrum (hail|scan) 02:52, 4 December 2009 (UTC)
I imagined that the purpose of SRAG would only be to approve and manage recruitment of editors. Although part of SRAG's job would be to evaluate study design so that studies with no hope of generalization don't waste valuable volunteer time, once the recruitment has been completed, there is nothing SRAG could do. What I really mean to say is this: You wouldn't need SRAG's approval to research Wikipedia, but you would need SRAG's approval to recruit editors via random talk page postings. --EpochFail(talk|work) 15:46, 4 December 2009 (UTC)

Recruitment process takes into account whether the editor has been recruited recently

Where is the data regarding which editors have been recruited recently, and how many times, going to be stored? Does recruitment mean invitation, because if it does, then the data can be stored somewhere public (it is already in the edit history of a talk page, but they can be inconvenient to search). If it means participation, then there may be privacy concerns that means that data ought not be publicly accessible - which brings me back to the original question.

In the same manner that jury duty often means you don't have to do it again for a while, you might want to consider adding a parameter to your opt-out template, expires=, and use that to exclude someone from recruitment for a period like so:

{{no research invitations|expires=2010-06-23}}

With that template parameter enthusiastic editors can opt to be eligible for another study if they so desire, or they can say "not right now, talk to me in a year" when they opt out.

Why is recruiting too many times a problem? Josh Parris 00:52, 4 December 2009 (UTC)

The issues you raise here are valid, but these issues would be addressed in the design and creation of the bot, which is beyond the scope of this policy/guideline which says that there will be a bot that will, in general, operate in this fashion.
Regarding recruiting too many times: Editors are primarily on Wikipedia to do wikiwork. To frequently ask them to participate in studies is a distraction to that task, and may turn them off to participating in them in the future. -- PiperNigrum (hail|scan) 02:52, 4 December 2009 (UTC)
Which is why I suggest leaving the duration between invitations up to the editor, they're best able to determine what frequency would turn them off future participation. Josh Parris 06:11, 4 December 2009 (UTC)
I think that this is a great idea and one that I've had as well. I imagine that we could always have a default frequency of requests/editor unless those editors specify via template call. Giving editors control over the frequency at which they can be notified will be crucial to getting this accepted by Wikipedians in general. --EpochFail(talk|work) 15:56, 4 December 2009 (UTC)

Uninvited participants

Is there going to be any mechanism to ensure that only invited editors participate in a research project? Why? If so: how? Josh Parris 00:57, 4 December 2009 (UTC)

In many studies, additional participants are usually welcome, as they suffer more often from lack of participants than too many. However, this would depend on the individual studies themselves. If they wish to allow only those who are invited to participate, it would be up to them to filter out data from other participants. If an editor chooses to participate in a study to which they are not invited and are allowed to do so, this will not impact their chances of being recruited to another study. -- PiperNigrum (hail|scan) 02:52, 4 December 2009 (UTC)
Assuming that the researcher doesn't want uninvited participants because they don't meet the selection criteria for the study, how is the researcher going to know if someone who turns up to the study was eligible/invited? I assume that part of the data acquisition would be collection of data to confirm the selection criteria were met? In other words, not a problem for WikiProject Research? Josh Parris 05:55, 4 December 2009 (UTC)
Correct. Wikipedia being a public resource, there is no way to insure that only the one being recruited will see the recruitment message. So, it is up to the study to determine if respondents actually fit their acceptance criteria. -- PiperNigrum (hail|scan) 15:34, 4 December 2009 (UTC)

If mass talk page recruitment postings are done without the approval of RAG, there are consequences

This doesn't need detailing; there are already policies regarding this (disruptive editing). What you might want to say is that "only SRAG consensus can drive RecruitmentBot, and RecruitmentBot is approved to make large numbers of edits to user talk pages - a researcher doing this themselves will expose themselves to administrative discipline for disruptive editing." Josh Parris 05:51, 4 December 2009 (UTC)

That sounds better to me too. I think you should make the edit. --EpochFail(talk|work) 16:00, 4 December 2009 (UTC)

Clarification of target of research

It occurred to be that "members of the community" could mean those reading, and not editing, Wikipedia. Canvassing those people is also possible: just whack a message on an IP's talk page. Or people could create accounts to change their viewing preferences, but never actually edit: again, they can be selected and messaged. Are these users within the scope of this proposal? Josh Parris 06:16, 4 December 2009 (UTC)

Most research to date has dealt with registered editors because IP addresses are not a reliable indicator of the actions of a single individual. However, it is certainly possible that future research may wish to include the readers of Wikipedia. -- PiperNigrum (hail|scan) 15:34, 4 December 2009 (UTC)
There is no public information that will tell us who is browsing Wikipedia at any given time, so we would be able to recruit non-editors by posting on their IP address talk pages. --EpochFail(talk|work) 16:03, 4 December 2009 (UTC)

I've made changes to clarify the distinction between readers, editors and users, hopefully making it clear that contacting any of the above for research is within SRAG's purview. Josh Parris 23:26, 8 December 2009 (UTC)

Opt-out template exists

Template:Bots is an established mechanism for controlling bots; when used to repel RecruitmentBot it would look like: 

{{bots|deny=RecruitmentBot}}

and we could extend the syntax to 

{{bots|deny=RecruitmentBot|recruit_from=2011-01-01}}

or perhaps 

{{bots|deny=RecruitmentBot|deny_until=2011-01-01}}

but the difficulty with that is that 

{{bots|deny=RecruitmentBot,SineBot|deny_until=2011-01-01}}

would have the deny_until part ignored by SineBot and as such may create confusion. Of course, RecruitmentBot could additionally use its own template, and perhaps only use that template for parameters - general opting-out would be covered by either the bots template or RecruitmentBot template, specific commands via the special RecruitmentBot template 

{{NoRecruitmentBot|until=2011-01-01}}

Josh Parris 02:26, 10 December 2009 (UTC)

It's good to hear there's an existing mechanism. It seems that the better approach at the moment would be to have a specific template for recruitment opt-out that might be able to utilize an existing template. -- PiperNigrum (hail|scan) 04:35, 10 December 2009 (UTC)
I agree, the last option would create a minimum of confusion. In addition, all bots are required to implicitly respect Template:Bots, so you'd get two control mechanisms. Josh Parris 06:37, 10 December 2009 (UTC)

Are there limits to group size, term length?

I'd suggest that

  • the minimum size be such that a timely discussion can take place regarding proposals (although what happens when you go below this "minimum" is beyond me; perhaps a recruitment drive?);
  • a minimum number of pure academics be mandated so that a number of expert opinions are included in all deliberations (although what happens when you go below this "minimum" is beyond me; perhaps a recruitment drive?);
  • a soft upper limit be placed on the size of the SRAG so that discussions don't get bogged down (perhaps the limit ought to be expressed in the length and duration of deliberations - if they're going too long, the group size needs cutting);
    • Why not just have these discussions be open to the public like WP:BAG? Then, it wouldn't matter how many members of SRAG there would be, because anyone can participate in the discussion. --EpochFail(talk|work) 16:16, 4 December 2009 (UTC)
  • that mandatory retirement of members be in response to the group reaching its maximum size; only active members need retire (weird, huh?) because they're the ones slowing things down.

Josh Parris 06:52, 4 December 2009 (UTC)

I think would could learn a lot from the bot approval group in this case. Although there are some good criticisms about how BAG works, it does work. I can't seem to find any information about upper and lower limits on the number of members in BAG though. --EpochFail(talk|work) 16:16, 4 December 2009 (UTC)

If the SRAG's role is to close discussions, I don't think you need all that many members, perhaps three active as a minimum so things get closed about the day they're meant to. All my previous blather no longer applies. And I don't see why you'd have an upper limit, because large numbers of potential closers will only have the effect of making discussions get closed earlier. Josh Parris 11:41, 11 December 2009 (UTC)

Agreed. I think it is reasonable to state the lower limit as a guideline and let SRAG worry about trying to follow it. --EpochFail(talk|work) 22:51, 11 December 2009 (UTC)

quid pro quo

Is it an explicit expectation of the policy that the results of research will be published; we'll let you take a chunk of our time if you educate us about ourselves? Josh Parris 04:28, 10 December 2009 (UTC)

With any study, there is no guarantee of publication in a journal or conference proceedings, but we could add a requirement that at least some publishing of results occur. If the results of the work are not published in any official publication, they must be published in some other "publicly accessible" way where "publicly accessible" means that the results can be retrieved without the permission of the author, although there could be a minor fee. A lot of journals and conferences will charge fees for papers to support their online repositories. I don't think we will ever be able to get around that problem. --EpochFail(talk|work) 18:26, 11 December 2009 (UTC)

Researcher as Wikipedian

It is preferable that the researcher's account have a prior history of positive contributions/interactions within Wikipedia, and is familiar with Wikipedia's policies and guidelines.

I'm concerned that this may introduce bias into the researchers. Is there are reason why this is desirable? What's wrong with someone who totally doesn't "get" Wikipedia interacting with its users? And, by the time they stumble across this guideline, surely they've come to some level of familiarity? Is it the PI that needs this, or the researcher doing the interaction? Josh Parris 04:50, 10 December 2009 (UTC)

I suppose that with this policy and SRAG, this is less of a concern than before. In the past, other editors tended to distrust researchers who had newly created account and went around asking for subjects for their studies, and showing them being a Wikipedian was to prevent some of those feelings. Apparently not everybody subscribes to WP:BITE. -- PiperNigrum (hail|scan) 20:21, 10 December 2009 (UTC)

Application form

I suggest you knock up a pro forma application form heavily annotated with HTML comments <!-- like this -->, and in the guideline have very clear step-by-step instructions in the manner of WP:AFD#How to list pages for deletion. You start with {{subst:pro forma research application}} and save that, and then fill out the various fields, then add it to some list somewhere. The application form can dictate all the required and optional fields, links to expanded discussions on them and so on. Once you go adding your application to the list, the SRAG will start its deliberations (some mechanism needs to exist to allow some time to get the paperwork completely filled out). You can ask for help by [[doing something or other]]. Josh Parris 05:07, 10 December 2009 (UTC)

Alternatively, Wikipedia:Editnotices seem a helpful way of doing this too. Playing around with both might be a good idea. Josh Parris 14:24, 10 December 2009 (UTC)

An example application form is planned, hence the example link the section on applying, we just hadn't gotten around to that yet. -- PiperNigrum (hail|scan) 20:21, 10 December 2009 (UTC)
I've knocked something up, see Wikipedia:Recruitment/Requests for approval/example Josh Parris 13:12, 11 December 2009 (UTC)

Study Recruitment Approval Group

Is it fair to say that the deliberations over studies are public, and any user can participate, but a SRAG member closes the discussion? And that any one member of the Study Recruitment Approval Group can direct the actions of RecruitmentBot? Josh Parris 05:13, 10 December 2009 (UTC)

Correct. Anyone on Wikipedia can come and participate in the discussion. SRAG will close the discussion and make the decision taking consensus of participants into account. -- PiperNigrum (hail|scan) 20:21, 10 December 2009 (UTC)
  • In that case, I'm going to draw a parallel between SRAG members and admins. They sit in judgement of the opinions voiced by advocates and determine the consensus. The advocates cite policies and guidelines relevant to the decision. At this time, there are no policies or guidelines to cite. This could turn out poorly.
Alternatively, we could just say it's a vote - but then you get all the problems associated with voting.
So, who gets to decide the criteria for approving a study? The process is specified, it's clear and it's reasonable.
Perhaps we could say, "editors contribute an opinion on how many people should be contacted to generate the requested participant numbers; people who don't value the research would say a low number (zero for example: I don't want this research to contact anyone)" or it could be split into Oppose/Support, with support needing to suggest the recruitment population.
This is a gapping hole.
  • Given that SRAG members close discussions, why the need to specify that they have any special attributes, or that there be skilled or knowledgeable in any area? What's special about them, and why? Josh Parris 09:03, 11 December 2009 (UTC)

SRFA and SAG

The redirect/shortcut Wikipedia:SRRFA Study Recruitment Requests for Approval and Wikipedia:SRAG Study Recruitment Approval Group are available. Josh Parris 09:43, 11 December 2009 (UTC)

Bot customization

Each study with unique criteria will require customization of RecruitmentBot; this will take an unknown period. This needs disclosing.

Can researchers provide their own customization of RecruitmentBot? Josh Parris 09:34, 4 December 2009 (UTC)

Sampling from Wikipedia editors is something that can be automated. These types of things would be in the configuration of RecruitmentBot and would not require code modification. In cases where the queries are overly complicated, the researchers could produce their own sample which would be approved for postings via the same failsafes (posting frequency, skipping those who've opted out, etc) as normal postings. --EpochFail(talk|work) 16:22, 4 December 2009 (UTC)
Having had some time to think about this, I think RecruitmentBot shouldn't be responsible for the selection of candidates, only recruiting them. It should take a list of candidates and parameters about how many the study wants, and work its way down the list contacting them according to the various rules about not bothering people. Some other tool - be it machine or human - can select the candidates, because:
  1. this is the hard part
  2. this is the part that changes for every study
  3. this is the part that the PI might want to do themselves
  4. it's the kind of thing the SRAG would desperately want to farm out to someone else
Josh Parris 23:26, 8 December 2009 (UTC)
Agreed. --EpochFail(talk|work) 23:01, 11 December 2009 (UTC)

I'm noted this distinction in the guideline. Josh Parris 08:17, 12 December 2009 (UTC)

Userbox

...the researcher will be allowed to place a template on their user page indicating that they have been approved to recruit subjects for their study on Wikipedia

Don't mention this. The recruitment message ought to carry all the authorizations and details. For example:

...Researcher's blather about saving the whales...
Wikipedia:WikiProject Research/SRAG reviewed and approved the Wikipedia:WikiProject Research/Approved/Whale preservation opinions of Wikipedia editors#Participant selection crieria for this academic study; you have been randomly selected on the basis of that criteria. Contact details for the researcher, study goals and other details can be found at Wikipedia:WikiProject Research/Approved/Whale preservation opinions of Wikipedia editors. I am a bot, and cannot answer any questions. I don't message users with {{goaway}} on their user or talk page; if I am malfunctioning, leave a message on my talk page.~~~~
It's certainly an option. I had envisioned that the recruitment message would be more study-specific, with a pointer to something more general indicating approval has been granted by SRAG. -- PiperNigrum (hail|scan) 04:35, 10 December 2009 (UTC)
The top part of the message would be detailed by the researcher in their application; I'd guess at the least it would include an external link and a short marketing spiel. The bottom part would be adjusted by the SRAG on a study-by-study basis, but I would imagine it would generally be that kind of boilerplate, perhaps in a box and a slightly smaller font - like a health warning. If the recruitment message is generally in this form, people would get to know what a valid invitation looks like; building legitimacy through familiarity.
The reason I propose this is to disconnect being a researcher from being a Wikipedian; the appearance of a userbox (or whatever) saying you're approved could easily be faked; the study approval ought to contain all the information regarding the study; a consistent process lowers the cost of finding out what's going on - if various studies have a multitude of ways of communicating the necessary information, a user can't just scan for the salient points. Josh Parris 06:03, 10 December 2009 (UTC)
I've been working on something like what you are proposing? --EpochFail(talk|work) 23:15, 11 December 2009 (UTC)
I see your idea, and raise you - only one link in the message (not the boilerplate) so it is clear what to do, and the information overload/disclaimer is put into boilerplate which is clearly delineated as such. Josh Parris 10:35, 12 December 2009 (UTC)

Acceptable research methodologies

... provide guidelines for acceptable research methodologies within Wikipedia ...

Forgive me, for I am but a layman, but did the guideline actually address this part of the intro under a different name? Have acceptable research methodologies been codified? Or is it the role of SRAG to determine what's acceptable? Josh Parris 04:24, 10 December 2009 (UTC)
That's a good point. Research methodologies has a specific meaning in the research community and I'm pretty sure that isn't what we mean to provide guidelines for here. What should be doing is providing guidelines for how research should interact with Wikipedia and the editors they study. --EpochFail(talk|work) 18:21, 11 December 2009 (UTC)

I've removed this from the guideline. Josh Parris 12:32, 12 December 2009 (UTC)

Proposal denied

The decision can be appealed upon resubmission of the application. An application that has been resubmitted and denied two times cannot be resubmitted again for consideration. A resubmitted application that has not been changed from its prior submission will be immediately denied.

Why two times? Why specify immediately denied? Be less proscriptive about this, and just do it that way implicitly. In fact, just go with:

The decision can be appealed upon resubmission of the application.

People smart enough to want to do research into Wikipedians aren't dumb enough to waste everyone's time trying to get approval to do so. They'll fix whatever problems are identified or give up. Josh Parris 00:59, 10 December 2009 (UTC)

While "two times" is arbitrary (and negotiable at this stage), the purpose is to indicate that there is a finite limit to the number of times SRAG will entertain applications that are repeatedly denied. I can certainly imagine some tenacious researchers who would keep trying to get a study in ad infinitum.
The immediate denial is to prevent a study that was denied from being resubmitted as-is, as its denial is likely due to some issue that would need to be addressed before resubmission. -- PiperNigrum (hail|scan) 04:35, 10 December 2009 (UTC)
My thinking is: when this becomes a problem, make the rule then. There's nothing stopping a SRAG member abruptly closing an application "this application is materially unchanged from a previous denied application". You have to give the judiciary some leeway and power, or gaming the system can become a problem. If a researcher becomes problematic, they're not likely got get SRAG approval anyway, and there's nothing other than admin powers to stop them being disruptive regardless of what "the rules" say. So, don't wp:bite the researchers, WP:AFG. Josh Parris 05:35, 10 December 2009 (UTC)
It may benefit us to rewrite the section to say, "Resubmissions of denied proposals without material changes will be denied without discussion unless the submitter provides justification for re-opening discussion." That would give SRAG members justification in closing re-applications quickly, but allows them to keep the discussion if there is merit. --EpochFail(talk|work) 23:07, 11 December 2009 (UTC)

I'm now going to come at this from a different direction. I keep getting the feeling that Requests need work up time; let's call them Proposals at this point. So the first thing a researcher does is create a Proposal, and SRAG members (specifically, but anyone in general) hold their hands as it's dragged up to an acceptable level. During this time, SRAG also forms an opinion as the conversion rate, or perhaps in the early days describes how they're going to decide when to stop inviting (because no-one knows what the conversion rate could possibly be, or what drives it). Once the Proposal is ready, it's shunted over to Requests, where the community at large either tears strips off it because "it's a terrible, terrible idea and why are you even bothering to make us read it", or they ignore it because it seems fine. A passed Request is enacted by SRAG.

SRAG decides what Proposals are put forward as Requests; the community gets a veto on that.

The effect of this is that only proposals with merit will be put before the community; other proposals will be stuck as proposals until they become acceptable to SRAG.

How does that sound? Josh Parris 07:42, 12 December 2009 (UTC)

I really like the idea of having a separate staging area for researchers to work their proposals into shape before they are turned into public "requests", but I am hesitant to let SRAG make the decision about when it is ready. It seems to me that the researchers could be responsible for making this call with input from whoever is interacting with them while they are organizing their proposal. The researchers have a natural disincentive to make sure they are ready so that avoid the whole public denial process and SRAG wouldn't be able to (and couldn't be accused of) sitting on a proposal for too long. --EpochFail(talk|work) 16:12, 14 December 2009 (UTC)

Recruitment completed

I hope that the researcher will communicate with SRAG and say "we don't want any more participants; we're now collating and analyzing data". At this point, should RecruitmentBot find its recruitment messages and remove them? Or annotate them with "This study has now closed"? Josh Parris 01:30, 10 December 2009 (UTC)

I think that if a study has received more participants than they needed from the recruitment messages, it would be up to them to filter out data from the additional subjects. Personally, I'd love more participants to a study than I was expecting, as it makes it easier to reach statistical significance in results. Having too few subjects has been more of an issue thus far than too many. -- PiperNigrum (hail|scan) 04:35, 10 December 2009 (UTC)

Okay, I presume then that in their application, the researcher would specify the time period in which they'd accept participants, and maybe even include this in their recruitment message. Imagine logging into Wikipedia after six months of inactivity and finding one of these messages; it really needs to communicate within itself when the recruitment ends. Josh Parris 06:34, 10 December 2009 (UTC)

Having an approximate length of study in the application would work. Also, the page(s) that the participants would visit in order to participate could indicate that the study is now over and no longer accepting subjects. -- PiperNigrum (hail|scan) 21:49, 12 December 2009 (UTC)
Having a notification added to a recruitment message that the study closed is only useful to anyone if it saves someone time/annoyance. I can imagine it would be very frustrating to finally follow that request to participate in a study only to find out, after a bit of reading, that the study is closed. Now, many studies need not close. For example, when we have run studies with interface modifications in the past, we have allowed anyone to join the study group and use the tool we are promoting/studying at any time whether they were invited or not. We'd then simply limit our analysis to the users we were interested in. For studies where this isn't possible, it seems to me that modifying or replying to the original recruitment message via recruitment bot is the least annoying action to take. --EpochFail(talk|work) 16:31, 14 December 2009 (UTC)

Study mechanism

I was imagining that researchers would conduct their data-gathering through a web-based questionnaire, accessed via a URL specified in the recruitment mechanism. Then I realised that we could specify a URL we control and redirect to the study's URL, and count the number of redirects we've made, using it as an approximation of how many participants the study has gotten.

Then I realised there are more ways to run a study than just a web-based questionnaire, and that this wouldn't work.

But: would most studies start this way? Could we mandate that they do, so as to gather these statistics to provide feedback on performance? Josh Parris 09:16, 11 December 2009 (UTC)

I think that this idea is interesting, and merits further consideration. As far as most studies starting with a questionnaire, that is not always the case. For example, we are currently running two studies NICE and HAPPI, which started with people installing and using our UI modification, with surveys occuring later. -- PiperNigrum (hail|scan) 22:33, 11 December 2009 (UTC)
Actually, if we use an internal page (like the one identified as the "read more" link in Wikipedia:WikiProject Research/Recruit2) then we can count it's traffic on a daily basis using http://stats.grok.se/ Josh Parris 11:43, 12 December 2009 (UTC)
I might be missing something, but how is this any more useful than just asking the researchers how many people have started/completed a survey? --EpochFail(talk|work) 16:48, 14 December 2009 (UTC)
This one's got upsides and downsides. Pageviews don't tell us a much as the researcher could, but on the upside we're not dependent on the researcher telling us and all the data is in a common format - it doesn't vary research to researcher. It would make a good raw statistic for prediction, but it isn't the one we're after. And it could tell us that, for example, the researcher's recruitment message is getting a lot of interest, but once they make it to a page that tells them what they're in for, interest drops away (and so for the next study we focus on "selling" the study harder in the internal page). There is no reason why we couldn't do both - ask the researcher for participant numbers, and measure click-throughs. Josh Parris 04:49, 15 December 2009 (UTC)

Don't panic

Collaborators! If you have been watching this page recently, you'll notice that I've moved a lot of content. That content was moved largely based on the discussions we had here and the structure borrowed from WP:BAG. You'll find content has been moved to the following pages:

I apologize if I have caused any frustration by taking on this task by myself.

--EpochFail(talk|work) 00:23, 15 December 2009 (UTC)

Not enough respondents

I expect a researcher will specify how many participants they want; the SRAG will use its expertise and experience to determine how many editors are contacted by RecruitmentBot.

If SRAG asks for 200 editors to be contacted, and 10 participate in the study, there needs to be a mechanism for the researcher to communicate with the SRAG and tell them that they've got 10 participants; please send another 60. And is that 60 a minimum, or an approximate number, or what? Josh Parris 01:30, 10 December 2009 (UTC)

A feedback mechanism to ask for additional recruitment does sound like a good idea, but its use ought to be limited. Researchers should understand that recruitment does not guarantee participation, and SRAG does not guarantee that they will be able to receive some minimum number of respondents, so we don't want to be recruiting repeatedly to get a specific number. -- PiperNigrum (hail|scan) 04:35, 10 December 2009 (UTC)
A study that doesn't get sufficient numbers isn't much of a study.
SRAG claims the authority to pester people, and expertise in recruitment. To build this expertise, SRAG needs feedback on its performance in recruitment, and it must analyze why an estimate it creates would be wrong. It has also got to record all of this to build up institutional memory. As such, one of the researcher's obligations must be periodic reporting to the SRAG of participation rates.
SRAG certainly can't guarantee a certain amount of partipants; but they could, for example, decide "we're not going to annoy more than 450 people to get your 60 respondents to determine if there's a relationship between the self-reported height of editors and their edit count", but in another study may decide that the limit might be 30000 editors (so, basically unlimited) to get ten respondents.
Would there be a problem with revisiting the recruitment process until the minimum number of participants are reached? In the early days, the SRAG is going to be feeling around in the dark anyway. Josh Parris 06:30, 10 December 2009 (UTC)
After some consideration, I think revisiting the recruiting process to send out additional waves of recruitment messages is a good idea. But, in addition to an attempt to reach the desired minimum number of subjects, I think there should be a ceiling to the number of additional waves of recruitment messages that can be sent. I'd rather not see, for example, the sending of 10 or 15 waves of messages for one study. Perhaps limited to 3 to 4 waves of messages. -- PiperNigrum (hail|scan) 21:17, 12 December 2009 (UTC)
It seems to me that this is something that needs a good set of guidelines rather than hard rules about the raw number of times additional recruitment messages could be sent. SRAG might, for example, not want to send additional recruitment messages if the study being performed has little value for the community and the conversion rate is very low. For studies that have high possible value to the community, SRAG might be willing to send several new recruitment messages to ensure that there are enough responses. It seems to me that decisions would need to balance harm to the community (consuming Wikipedians' wiki-work time/effort) with the likely value of results to the community/humanity. --EpochFail(talk|work) 16:23, 14 December 2009 (UTC)
I think this is the right approach; avoid being proscriptive, particularly in the early days. Josh Parris 10:33, 15 December 2009 (UTC)
Don't get too hung up on "waves" of recruitment. Each editor is a distinct individual who either gets contacted or doesn't. If the bot is operating slowly enough, response rates can be monitored and the size of the invitation group can be adjusted. And besides, once one clump of invitations have gone out, SRAG will have a decent idea of the response rate and judge if more people ought to be bothered. Josh Parris 10:33, 15 December 2009 (UTC)

Types of research

This section needs expanding, even with bad examples, because there's absolutely nothing to go with here. Josh Parris 23:46, 12 December 2009 (UTC)

 Done --EpochFail(talk|work) 20:20, 15 December 2009 (UTC)
 Done moar --EpochFail(talk|work) 23:12, 17 December 2009 (UTC)

Confirmation of Institutional review board approval

Regarding contacting the Institutional review board to confirm approval by phone or email: Is anyone ever going to phone? Generally, disclosing email addresses on web pages is discouraged due to harvesting by spam-bots; will researchers disclose their email? How will the SRAG know that they are calling the cited institution, and not the researcher's mate Steve? And why is IRB approval needed by academics (in the US) and not, say, corporate researchers? Josh Parris 14:20, 10 December 2009 (UTC)

As you can see from the IRB article, their approval is needed for human subjects research, to ensure that the research does no harm to the subject. For the most part, the kinds of research I'm familiar with about Wikipedia are granted exempt status, indicating that the study does not need further IRB oversight, but it's up to the IRB to determine this. To my knowledge, IRB's exist only in academic institutions (somebody out there correct me if I'm wrong). -- PiperNigrum (hail|scan) 20:21, 10 December 2009 (UTC)
  • Whose problem is it if the IRB hasn't approved their research? Surely the researcher? I'm just wondering why academics funded by the US government have an extra hurdle to jump than the other researchers this proposal covers. Why do we care about IRB approval, surely the SRAG will individually assess each study on its merits? Josh Parris 08:37, 11 December 2009 (UTC)
    • In academia, in the US, you can't publish without gaining IRB approval before performing a study. If researchers require IRB approval and don't have it, I can't see why they should be approved to start recruiting. This IRB approval will be another piece of information that concerned Wikipedians can use to determine if a study should be accommodated by Wikipedia and its editors. --EpochFail(talk|work) 16:45, 14 December 2009 (UTC)
    • ♦ I agree with Josh -- I don't think that the SRAG should worry about IRB approval. In particular, requiring IRB would exclude independent researchers, which seems to go against the spirity of Wikipedia. I have a more detailed proposal below. --R27182818 (talk) 22:59, 20 December 2009 (UTC)
  • Regarding contacting the Institutional review board to confirm approval by phone or email: Is anyone ever going to phone? Generally, disclosing email addresses on web pages is discouraged due to harvesting by spam-bots; will researchers disclose their email? How will the SRAG know that they are calling the cited institution, and not the researcher's mate Steve? Josh Parris 08:49, 12 December 2009 (UTC)
    • Most IRBs will have their contact information listed publicly already. If you don't think you are calling the listed institution, you'd be able to easily find their phone number yourself through their website or a third party like the institution they are part of. (See the University of Minnesota's IRB site) --EpochFail(talk|work) 16:45, 14 December 2009 (UTC)

Copyedit, minor reframing

♦ I made some copyedits and framing changes directly on the page, in the spirit of boldness. Please feel free to revert in part or in full if these seem wrong. In particular:

  1. I dialed back the emphasis on "protection" and "harm" still further, though my changes were minor. This is the result of a F2F discussion with User:EpochFail, which I summarize as follows: Pedantically, unwanted research solicitations on one's talk page are indeed nonzero harm. However, I believe that harm has negative connotations which are excessive for this use. Similarly, I believe that protect implies that the person or thing being protected against has unsavory or malicious intent, which is not true of the scientists studying Wikipedia. Thus, I believe the use of these terms is an unnecessarily negative framing of Wikipedia researchers and science in general. I used terms like mediate, respect, and preferences.
  2. I also replaces editor with community member or user, as researchers are interested in all Wikipedians, not just those who edit.
  3. I replaced subject with volunteer and participant as I believe the former is academic jargon with negative connotations among the audience for this page.

Comments welcome, of course. --R27182818 (talk) —Preceding undated comment added 22:56, 20 December 2009 (UTC).

Proposed change to SRAG goals

♦ I am concerned with SRAG's responsibility to make value judgements about specific research studies. In particular, I believe that all of the examples currently given in the Types of research are potentially meritorious. I propose instead that SRAG take responsibility simply to ensure that research studies are accurately described to (potential) participants. Specifically, here is proposed wording for SRAG's two goals:

  1. Ensuring that individual community members' preferences on whether and how frequently they are invited to participate in research studies are respected.
  2. Ensuring that participating in each research study is fully and accurately described, including but not limited to (a) who is conducting the study, (b) who else has approved the study, and (c) what participating would involve.

Determinations of risk to users are up to the IRBs, IMO, and individual users can make their own decision about whether they wish to participate. --R27182818 (talk) 23:09, 20 December 2009 (UTC)

♦ Following up, it seems that perhaps my proposal is too strong. In particular, I don't believe we should allow marketing research (e.g., "Do Wikipedia users prefer Coke or Pepsi?"). Following in the style of the Debian Free Software Guidelines, how about some tests:
  • Is the nature of the proposed research adequately disclosed?
  • Is the proposed research scholarly?
If the answer to both tests is yes, then it's approved. Or, if you believe that my proposal is too strong, here are alternate tests:
  • Is the nature of the proposed research adequately disclosed?
  • Does the proposed research respect both Wikipedia community members and Wikipedia community norms?
This still avoids the value judgement which concerns me. --R27182818 (talk) 23:46, 20 December 2009 (UTC)

Archiving threshold lengthened

♦ I'm finding it hard to follow the discussions here as a peripheral participant. Thus, I increased the archive threshold to 14 days. --R27182818 (talk) 22:21, 20 December 2009 (UTC)

♦ Or not -- it seems not to have taken. Please, someone smarter than I fix it? --R27182818 (talk) 22:23, 20 December 2009 (UTC)
As an active participant, I was finding it hard to follow the discussions here when there were 22 discussions that had not been touched in a week+ that were cluttering up the page. However, since we've begun breaking off separate articles for things like SRAG and the applications, talk pages should be more managable. So, I suppose we'll give 14 days a whirl. -- PiperNigrum (hail|scan) 15:13, 21 December 2009 (UTC)

Violating community norms

Sometimes research needs to violate community norms. Why are we asserting that such research will not be allowed before it is even considered? --EpochFail(talk|work) 18:33, 21 December 2009 (UTC)

I agree, if the researcher proposes to violate norms with good reason, SRAG ought to be able to deliberate on the matter. Josh Parris 05:37, 22 December 2009 (UTC)

Stagnating

This discussion is stagnating and needs more people brought in. I suggest a short note on WP:VPP to get some policy wonks in here. Josh Parris 12:12, 5 January 2010 (UTC)

It's currently the slow season in US academia—winter break between fall and spring semesters. We've just started to spread the word to get more input, starting with the wiki-research-l mailing list. Village pump is likely for the next pass. -- PiperNigrum (hail|scan) 20:53, 5 January 2010 (UTC)
Piper and I are worried about having too many conflicting ideas bouncing around before we had a coherent proposal in place. I'm in favor of waiting for a wave from wiki-research-l and foundation-l before we go to WP:VPP for more feedback. --EpochFail(talk|work) 23:34, 6 January 2010 (UTC)

Issues

A brief read raised just a few questions in my (jameshowison) mind:

1. Seems to be a little conflict between two sentences in the Publication section:

"or it could mean that the results will be submitted to a scholarly publication where the papers can be retrieved for free or a minor fee." and "Research published only in for-fee publications will not be permitted."

Are you intending a distinction between "minor fee" and "for-fee"? Or is it the case that its ok if it goes to a "for-fee" publication (of any amount) as long as it is also freely available (known as Green Open Access). One could imagine negotiating that with a "for-fee" journal; in fact this policy might give us some minor leverage.

2. Has there been any discussion of how to enforce the one year provision? Just "Expect negative consequences if nothing is published/made available"?

3. There wasn't much discussion of how the deliberation process would unfold. Would it be worth elaborating on that? Perhaps that it be done publicly visibly (as much as possible?

4. Would SRAG ask for a copy of the institutional review board certification (human subjects review) or perhaps the other way around? Jameshowison (talk) 21:22, 5 January 2010 (UTC)

  1. Many ACM/IEEE publications are only available for a "minor" fee, and authors frequently place copies on their own websites. The intention here is that the results be published in a way that is publicly accessible.
  2. No specific discussion of enforcement has been made yet. I would expect that they would certainly not be likely to have future SRAG requests approved. One year is a "soft" deadline, which can be extended by petition. This is intended for the results to be published within a "reasonable" timeframe, with the understanding that research often does not run on a schedule.
  3. As with many discussions on Wikipedia (AfD's, Bot approvals, etc.) they would be posted and any Wikipedia reader will be able to participate. The same policies about publicizing these discussions would apply.
  4. Again, no specific implementation details yet. The presumption is that the IRB would have some information (study name/id number?) available for us to see, and we would have phone/email contact if SRAG should need to contact the IRB to see that yes, someone did think that hitting editors over the head would cause them no appreciable harm. :)
-- PiperNigrum (hail|scan) 21:48, 5 January 2010 (UTC)
2. This will be a hard one to enforce, but I'm in favor of keeping the requirement so that it can be enforced during future petitions to SRAG by the same researcher. --EpochFail(talk|work) 23:30, 6 January 2010 (UTC)
1. This is a good point. I have updated the article to make it consistent.--EpochFail(talk|work) 23:30, 6 January 2010 (UTC)

Open access / sharing findings

First off, when is a study considered complete? After data collection, analysis, or when funding runs out? Also, while 12 months from completion of study seems like a long time (especially for you lucky ACM folks), that can be rather short in some disciplines. It is not unusual for an excellent, final paper to take that long just to get through peer review. And that doesn't count the time a paper may sit in a queue, as some journals will only publish a certain number of articles per issue. This becomes a problem because many of these same disciplines also have very strong norms against preempting the journal, even in part. The NIH's open access requirement is 12 months from publication, and I think that is becoming the standard. Stu (aeiou)I'm Researching Wikipedia 19:47, 6 January 2010 (UTC)

Staeiou, you bring up some very good points about the requirements for sharing results. When we were drafting that requirement, we worried about some of the same issues you brought up, but we felt it was important that there was some requirement for publication so that value would be given back to the Wikipedia community. Our intention was to fit most cases but leave room for reasonable circumstances like those publication systems that require (relatively) excessive amounts of time before publishing results.
You mentioned the NIH's requirement for open access 12 months after publication as being a standard we could adopt, but I worry about those studies that are conducted, but never make it to publication for whatever reason. It seems to me that the community should have access to results in a "reasonable" amount of time regardless of whether or not the results are published in a journal, conference, etc. --EpochFail(talk|work) 23:14, 6 January 2010 (UTC)

Definition of Scope

♦ We need to define the scope of SRAG's authority, i.e. what WP research needs to be approved and which is exempt. For example, soliciting users "clearly" needs to be approved, while analyzing dumps doesn't. I propose:

Research which (a) contacts Wikipedia users using Wikipedia tools or (b) manipulates the live Wikipedia site must be approved by SRAG.

So, HAPPI would need approval, but running something like HAPPI in a lab study where volunteered were solicited outside Wikipedia wouldn't. Manipulating WP articles would, but showing people modified or wrapped articles wouldn't (assuming the modification wasn't done on Wikipedia itself and subjects were recruited outside WP).

Specifically, I think there needs to be a formal request to publication venues and reviewers that research which should have been approved by SRAG but wasn't be rejected. --R27182818 (talk) 23:52, 20 December 2009 (UTC)

I don't believe there's any mechanism at the moment for authorizing JavaScript add-ons, which HAPPI is. Asking people to use HAPPI manipulates the live Wikipedia site, so you could combine both into the one:
Research which changes the live Wikipedia site must be approved by SRAG.
But for that simpler statement: what about researcher created bots? Aren't they already covered by WP:BAG?
And: what about emailing users? In general, should SRAG prohibit email contact, given that there's a very strong taboo against using it (even AfD nominations won't email a user)? Or do we just leave that for case-by-case deliberations? Josh Parris 05:59, 22 December 2009 (UTC)
I think users should be well defined as well. What is meant by users? contributors - readers?Bilalak (talk) 20:46, 5 January 2010 (UTC)
  • Short answer: yes
  • Long answer: Earlier drafts of this proposal used community member and editor, but it was pointed out that these terms were too specific, and we should encompass both editors and readers of Wikipedia under the recruitment policy, hence the use of the more general term user.
-- PiperNigrum (hail|scan) 15:02, 6 January 2010 (UTC)
See Wikipedia:Reader Josh Parris 01:20, 9 January 2010 (UTC)
I think that we should say that researchers who "interact with editors or readers on Wikipedia in order to collect data" need to go through SRAG. This would include server-side modifications (which also need technical approval, right?), interviews, surveys, participant observation, etc. However, I don't think SRAG approval is needed if I put up fliers on campus to interview students who read or edit Wikipedia, e-mail users who have put their e-mail addresses on their userpages, or put the HAPPI software on local computers and had students use it. This wording would also make it okay for someone interested in researching Wikipedia to test the waters before deciding to collect data. Stu (aeiou)I'm Researching Wikipedia 20:28, 6 January 2010 (UTC)

Ethnography and recruitment

I think we need to distinguish between ethnography and interview/survey methods. (I also think I need to work on the Ethnography article, but I don't want to get into an academic discussion). Obviously, interviews and surveys need to recruit subjects, obtain consent, and get SRAG approval. Participant observation is a far trickier matter, because subjects cannot be recruited in the traditional sense. Still, all researchers who interact with Wikipedians in order to obtain data need to go through SRAG, and this includes ethnographers and participant observers.

As someone who uses these methods to study Wikipedia, this is something I struggled with for some time. When most ethnographers enter a community or group, they typically first obtain permission from the leaders, who announce that there is ethnographer roaming around. Then, the researcher is expected to identify themselves when they meet new people. This doesn't cleanly transfer to Wikipedia, so my solution was to append the "I'm researching Wikipedia" notice to my signature. I also wrote up a short document that said who I was, what I was doing, and that I'd go away if you want me to and would protect your privacy if we talk off-wiki. This may be overkill (a lot of Wikipedians thought it was), and I doubt we will have many ethnographers, but it is worth discussing if such disclosure is necessary for all ethnographers. Stu (aeiou)I'm Researching Wikipedia 20:08, 6 January 2010 (UTC)

The working draft currently states: This is the only way researchers are allowed to initiate' contact with Wikipedia community members.
I'm not sure that this statement agrees with the scope I think "recruitment" should have within this proposal. I feel as through, when this policy refers to recruitment and SRAG, it should only refer to those types of research methods that require recruiting samples of users via talk page postings. On the other hand, I'd like to see an additional section added to this policy to describe acceptable approaches for ethnographers such as yourself. If you are interested in working on such a change, I'd be happy to help. --EpochFail(talk|work) 23:22, 6 January 2010 (UTC)
Now that this is spread out over a half dozen pages, I can't find it - but I recall that one of the things the SRAG was proposing to cover was intentional vandalism. There's no recruitment or interaction involved there. Josh Parris 09:50, 9 January 2010 (UTC)
I thought that the idea for SRAG to cover anything other than recruitment requests had been dropped. --EpochFail(talk|work) 16:02, 11 January 2010 (UTC)
Okay, I've split interview/survey and participant observation, and basically written participant observers who don't survey or interview out of the requirements. They are recommended to consult with the SRAG though. Stu (aeiou)I'm Researching Wikipedia 08:25, 21 January 2010 (UTC)

Non-compliance with SRAG

First, we need to really publicize this on wiki and in academia once we finish. However, there will be researchers who just don't know about this process, and they shouldn't be punished for that. I think we should politely warn researchers who begin unauthorized research and encourage all editors and admins to do the same. Push them into the SRAG with no hard feelings if they go through the process.

I don't like the idea of trying to blacklist research that should have gone through the SRAG but didn't. I completely agree that we should make noise if the researcher is told to go to SRAG but doesn't, or if SRAG denies a study but it is done anyway. Again, there is bound to be good faith research done in ignorance of this policy, and it shouldn't be punished. Stu (aeiou)I'm Researching Wikipedia 20:43, 6 January 2010 (UTC)

At one point, we had exactly this type of language inside of the article, but removed it for reasons I cannot recall. Is it common to prescribe punishment/non-punishment within policy articles like this one? --EpochFail(talk|work) 23:26, 6 January 2010 (UTC)
It was changed because a different wording was suggested. Wikipedia talk:Research/Archive 1#If mass talk page recruitment postings are done without the approval of RAG, there are consequences -- PiperNigrum (hail|scan) 23:34, 6 January 2010 (UTC)

Disclosure requirements

Hi! This seems like a fine idea. However, I'd like to see a bit more detail about disclosure requirements, rather than leaving it open. Generally, I'd expect any formal academic study involving human subjects to provide the contact details of someone to make inquiries about the research (a supervisor, where appropriate, other where not), whether or not it has been approved by the ethics committee, (acknowledging that in most institutions it would have to go through an ethics committee or review board for approval) and a statement of how the privacy of participants will be respected (that would normally be part of the ethics approval process, of course). Forgive me if this has been covered elsewhere, but I've witnessed a number of research projects on Wikipedia where it wasn't clear who to contact about the research, the methodology was significantly flawed (in such a way to suggest that I'd question whether or not approval was given), and where I had concerns about the possible identification of participants that weren't adequately addressed. - Bilby (talk) 02:56, 12 February 2010 (UTC)

The page on the Subject Recruitment Approvals Group has some information about this. One of the line items in an application for recruitment approval is to provide IRB information. However, the purpose of this policy is not to approve their research or their methods – that's their IRB's job. This policy is more related to if/how the researchers can contact Wikipedia users to ask for their participation, and the process by which that recruitment occurs. -- PiperNigrum (hail|scan) 20:15, 12 February 2010 (UTC)
I feel as though most of the concerns you've raised have been touched on in the Requirements for SRAG approval section, but the need for information about privacy within the requirement of informed consent could be explicit. As far as methodology, I worry that people often get the impression that the researchers don't know what they are doing because those researchers can't publish their complete methodology until after the experiment has been completed. For example, if a subject in a control group knew that they were in a control group, the results related to them would be compromised. --EpochFail(talk|work) 20:51, 12 February 2010 (UTC)
Fair enough. :) I see now that those issues are mostly covered in the application form - but it still seems handy to spell out the requirements here, as there is a discontinuity between the two. This page talks about very general issues, but I'd like to see a bit more about specific requirements, on the grounds that I'd look at this page, figure out if my study is ok, then I might do some work on it before tackling the application form. And then I'd find the additional requirements, and would need to guess as to which page has priority. Can I do research if it meets the requirements here, but doesn't address everything on the application form? Or are the items in the application core? (I'd add that my own research doesn't fall under the scope of what is being covered here, so this isn't really a concern for me personally).
On the methodology front, I understand what you mean about only seeing part of the process, and I agree. I don't feel that the full methodology should be described here. :) My concern isn't that it be shown, but that I've seen instances where the researcher hasn't provided the information I would expect, and where their methodologies were seriously flawed to the point of being unpublishable. Which led me to believe that they were misrepresenting why and for whom the research was being conducted. So I like the process being proposed, and fully support it, as it would address those instances. - Bilby (talk) 00:10, 14 February 2010 (UTC)

Typo?

"This policy describes how researchers can recruit members of the via talk page postings..." Members of the what? Steven Walling 19:03, 2 March 2010 (UTC)

Thanks for catching that.--EpochFail(talk|work) 23:39, 2 March 2010 (UTC)

Making the raw data also open?

In practice, it is easy to make the results accessible for no fee by submitting a technical report e.g. to Arxiv before sending the publication to a journal / conference. A more interesting requirement, that would have huge benefit to the research community, would be the one of making the raw data available to all for free. So for instance if you do a study on whether people prefer revision A or B of articles, for many A-B pairs, the raw data results would be available to all. In this way, the volunteer time that went into an experiment could be reused for subsequent investigations. Of course, the data would have to be anonymized; in general, IRBs would have to approve of the raw data disclosure along with the rest of the project. We could have a wording that says "Once a study is completed, and the results are made available in publications, in three (or six?) months the authors will make available the raw data that was gathered as part of the study, after anonymizing it so that the identity of the participants is not disclosed". Or something to that effect. Lucadealfaro (talk) 19:44, 2 March 2010 (UTC)

♦ In principle, I agree; however, I worry that acceptably anonymizing the data will be prohibitively difficult in some cases and have a chilling effect - in other words, I see a tradeoff between more research with no raw data requirement and less research with a raw data requirement. I'd support a proposal to encourage release of suitably anonymized raw data but not one to require it. --R27182818 (talk) 22:27, 2 March 2010 (UTC)


RFC: Proposal -> Policy

Resolved
 – Upgrade to policy. If this is disputed, please use this talk page to hash out issues Hipocrite (talk) 20:32, 7 April 2010 (UTC)


This policy should be implemented for the following reasons:

  1. To allow important research of Wikipedia and its users to take place.
  2. To ensure that that individual community members' preferences on whether and how frequently they are invited to participate in research studies are respected.
  3. To ensure that research studies are respectful of participating community members and community norms.

I've notified all of the active members of Wikipedia:WikiProject_Research via talk page postings. --EpochFail(talk|work) 15:49, 2 March 2010 (UTC)

Note that this policy has a sister approvals group that mediates the recruitment processes mentioned in this proposed policy. —Preceding unsigned comment added by PiperNigrum (talk • contribs) 16:30, 2 March 2010 (UTC)

  • Support Wikipedia will benefit with this way to facilitate and control research into those who edit and use it. This policy provides a framework for researchers to operate in and a mechanism for Wikipedians to stop researchers contacting them. Josh Parris 15:07, 3 March 2010 (UTC)
    • Note: I was involved during the development of this policy, but have no vested interest in its outcome. Josh Parris 12:31, 19 March 2010 (UTC)
  • Support: This policy will help researchers carry out their research in a way that benefits Wikipedia, while respecting the wishes of members of the community. The needs of Wikipedia are carefully protected, while offering a way for researchers to carry out their studies. --Riedl (talk) 16:48, 12 March 2010 (UTC)
  • Support -- Good idea overall. Can lead to more and better research, and less annoyance. Maurreen (talk) 21:43, 12 March 2010 (UTC)
  • Support This proposed policy and SRAG would offer both community control over when researchers get to recruit and give researchers legitimacy when they recruit subjects for surveys, interviews, UI testing, etc. --EpochFail(talk|work) 15:34, 15 March 2010 (UTC)
    • Note that I wrote a significant portion of the proposed policy. --EpochFail(talk|work) 15:32, 18 March 2010 (UTC)
  • Support, per above and personal experience helping researchers deal with the unpleasant experience of trying to recruit volunteers without this policy. Note: I helped craft this proposal, so keep that in mind when "counting" my not-vote. --R27182818 (talk) 14:35, 17 March 2010 (UTC)
  • Support; the proposal suggests a framework that will improve transparency for the community, as well as making the process for researchers straightforward. I find both to be a good improvement over the current situation. Note: I have participated peripherally in the development process of the proposal. --Nettrom (talk) 18:31, 23 March 2010 (UTC)
  • Support. As a researcher, I'm especially sensitive to muddying the waters of an existing community (or that other researchers will give the rest of us a bad name). Even the most respectful attempts at recruiting can come across as spammy; this policy and the accompanying recruitment process will go a long way toward establishing best practices. More legitimacy for researchers, and less nuisance for editors. Grammarnerd (talk) 19:50, 23 March 2010 (UTC)
  • Support A research policy for Wikipedia would be really positive for both academics and community members. Wikipedia has acquired a great momentum as a flagship collaborative project in the Internet, thus attracting the attention of a growing number of scholars and researchers. However, we oftenly see how the candid attempts of researchers to gather information from members of online communities render to undesired situations, annoyances and misunderstandings. The same has happened already in other big and prominent online communities such as Apache, Gnome or Mozilla. This policy would allow to respect the legitimate right of users who do not like to be bothered with studies and surveys, whereas interested volunteers can help researchers in their challenging task of understanding Wikipedia in a better way. GlimmerPhoenix (talk) 13:26, 24 March 2010 (UTC)
  • Support. Having a defined research policy for Wikipedia is beneficial for both the community and the researchers interested in phenomenon within Wikipedia. Several papers and applications have been developed by researchers that could benefit Wikipedians in fostering community, getting needed work done, and improving policy. This seems like a clear win-win relationship, and having a clear policy could help researchers be as non-intrusive and respectful of the community as possible while still providing a great benefit. CMUResearcher (talk) 19:36, 24 March 2010 (UTC)
  • Support. While the flareups are probably limited to a few vocal users, these problems can be very discouraging to worthwhile research, and researchers are left to fend for themselves in trying to defend their actions. Having a clearly-defined policy on research canvassing, with a community-blessed mechanism for making contact with potential research subjects, will make it easier to do good research while simultaneously encouraging transparency and open publication in the research community. Elehack (talk) 15:03, 25 March 2010 (UTC)
  • Support I stumbled on this page and think it is a good idea. Self-discovery (can that be applied to a website?) is a good thing and this project seems like it will be a good step in that direction. TNXMan 18:54, 29 March 2010 (UTC)


Scope seems too narrow in a couple of regards

♦ The proposal contains the following core sentence: "This is the only way researchers are allowed to initiate contact via talk page postings to a large group of Wikipedia community members." Two thoughts. First, shouldn't SRAG regulate solicitations via other methods - for example, can't one e-mail users by some link rather than posting on talk pages? What about contacting WP users using a list of e-mail addresses one has from some other source? Second, what's the definition of large? Am I exempt from SRAG if I only need to solicit 2 users via talk page? 5? 10? 20? IMO, "to a large group of Wikipedia community members" should simply be struck. --R27182818 (talk) 22:34, 2 March 2010 (UTC)

If we strike the "to a large group of Wikipedia community members", we are left with 'This is the only way researchers are allowed to initiate contact via talk page postings' and then I'm not allowed to post on anyone's talk page without SRAG approval. Although I agree that "large" is hard to define, I think it is important that researchers are allowed to interact with community members without going through SRAG except for extreme circumstances (eg. mass recruitment). --EpochFail(talk|work) 22:47, 2 March 2010 (UTC)
♦ Well, what about striking that text and expanding what "initiate contact" means - initiate contact for the purposes of recruitment? Or, strike the text and add some exceptions? Maybe you can give some examples of when you think approval is not necessary. If nothing else, "large" needs to be defined. I feel that this is a situation where researchers have incentive to interpret this sentence more generously than SRAG would like. --R27182818 (talk) 16:23, 3 March 2010 (UTC)

<--outdent-- How about something like the following:

A researcher must obtain SRAG approval for any talk page postings where:
  • The researcher has not had prior communication with the user(s) being messaged
  • The posting is not related directly to work on Wikipedia content, policy or systems.
  • The posting is not personal communication.
  • The posting was not solicited by the user.
  • The user is not a member of WikiProject, work group, etc. managed by the researcher.
  • The user is not or has not recently become a subject of one of the researchers studies.
As rule of thumb, if a researcher wants to post an unsolicited message on the talk page of an individual with the intent of collecting data from them for use in a study. That posting will require approval from WP:SRAG.

As you can see, I've dropped "large" in favor of being explicit about the types of message being posted. --EpochFail(talk|work) 19:18, 3 March 2010 (UTC)

♦ I like it. How about the following modification (this also addresses my first concern):
A researcher must obtain SRAG approval to use tools within Wikipedia (e.g., talk page postings) to contact Wikipedia users when all of the following apply:
  • The communication would be the first by that researcher and the user (automated talk page postings by bot do not count as prior communication).
  • The message is not personal communication.
  • The message is not related directly to work on Wikipedia content, policy, or systems.
  • The user did not solicit the message.
  • The user is not a member of a WikiProject, work group, etc. managed by the researcher.
  • The user is not a current or recent participant in one of the researchers' studies.
As a rule if thumb, if a researcher wants to contact an individual with the intent of collecting data for use in a study, and that contact is unsolicited, he or she will need approval from WP:SRAG. --R27182818 (talk) 16:06, 4 March 2010 (UTC)
Agreed. I've added this within the current structure of the article. --EpochFail(talk|work) 22:44, 5 March 2010 (UTC)

Making a policy rfc

The RFC filed on this page didn't take the relevent steps. Firstly, it should have been listed at Wikipedia:Requests for comment/Wikipedia policies and guidelines. Further, for the creation of new mandated bodies with real powers, it should have been more widely advertised. As such, I decline to state that a broad consensus exists to make this policy. If you need help drafting or advertsing this proposed policy more widely, I am happy to help. Hipocrite (talk) 15:25, 7 April 2010 (UTC)

The RFC was listed in Wikipedia:Requests for comment/Wikipedia policies and guidelines, but since it had been open for over a month, it has been de-listed by RFC bot. The advertising that I have participated in includes the following:
I also have evidence that our advertisements has been successful. View statistics for the proposed policy and its talk page show an activity spike that starts on March 2nd, the day the RFC was first posted. This suggests that the RFC was initially successful in attracting viewers. If this is not sufficient advertisement, please advise us towards how to increase participation because we are honestly at a loss. --EpochFail(talk|work) 18:53, 7 April 2010 (UTC)
Oh. Sorry. Revising. Hipocrite (talk) 20:31, 7 April 2010 (UTC)

Regardless of my personal reservations, which are strong, there is clear consensus that this does not hurt. Hipocrite (talk) 20:36, 7 April 2010 (UTC)

Your bump is less than 100 views and you are claiming that is enough to make something policy? I think not. A 2% increase in wikipedia's policies is too high a cost for something that really really doesn't need a policy to deal with it.©Geni 02:11, 8 April 2010 (UTC)

Probably because of web crawlers and mirrors, the number of page hits in the stats.grok.se link (333) is roughly the minimum monthly hits that any WP-space page will get; that suggests that not many people have looked at the page yet. I see the policy tag has been removed, and I don't have a position on that. The page makes sense to me, I can see this as a useful policy some day, but my guess is that putting the policy tag on before people are ready for it will generate opposition. - Dank (push to talk) 02:46, 8 April 2010 (UTC)
Comparing the numbers for WP:Research vs WP:BAG, there is a noticeable difference in their number of views. That suggests that it has received more attention than just bots and crawlers. No policy is going to appeal to everyone, but removal of a policy tag that an independent editor deemed was appropriate seems to not be the right thing to do -- particularly by someone who had not previously participated in the months of discussion leading up to this. -- PiperNigrum (hail|scan) 02:57, 8 April 2010 (UTC)

Nowadays promotions to policy, even guideline, status require much more than that. You need to link a RFC at WP:CENT, advertize at WP:VPR, WP:VPP (with a clear header "proposal to promote wp:research to policy status"), etc. I have looked at the users who participated in the prior discussion, and most of them are involved in the development of this, the poll is not representative of the community. You'd have had much more people commenting if it had been properly advertized, you need to start a new one. Moreover, I don't see why this needs to be a policy. Cenarium (talk) 16:44, 8 April 2010 (UTC)

♦ Thanks for the suggestions, Cenarium. As for why this should be a policy, please take a look at the extensive discussion in the archives of this talk page. --R27182818 (talk) 17:16, 8 April 2010 (UTC)

Where is the problem?

After reading this, I have to say this is WP:CREEP.

  1. Are there cases where researchers canvassing for subjects have caused problems?
  2. If the researchers act as users, they are part of the community, and have to live with the same rules and risks as everybody else, so this policy would be redundant. If they don't act as users, this policy is pointless to begin with.

Regards, Paradoctor (talk) 07:42, 6 March 2010 (UTC)

♦ Yes, see Wikipedia_talk:WikiProject_Research#Examples_of_unmediated_interactions. There have been several instances when studies have been derailed because of misunderstandings between researchers and the community. --R27182818 (talk) 16:57, 6 March 2010 (UTC)
The problem this policy tries to address is that "it is important to respect the wishes of Wikipedia community members on whether and how often they are contacted in this way". I won't let a committee decide that for me. What you can and should do is have a setting added to user preferences, or some user templates, e. g. {{i'm a guinea pig}}/{{use animals not wikipedians}}/{{property of SRAG}}. Paradoctor (talk) 21:48, 6 March 2010 (UTC)
The committee does not decide that for you. The policy is that you will have the ability to control whether or how often you would like to be contacted to be asked to participate via either {{bots}}, or a SubjectRecruitmentBot-specific template. -- PiperNigrum (hail|scan) 22:21, 6 March 2010 (UTC)
"The committee does not decide that for you.": Hm? How does that mesh with "Subject Recruitment Approvals Group (SRAG), a public discussion group, to control who will be allowed to make those postings"? Paradoctor (talk) 22:45, 6 March 2010 (UTC)
SRAG is the body that mediates the process of approving/denying researcher's requests to make postings to recruit users, and controls the SubjectRecruitmentBot that actually does the posting of messages. The determination of who receives these messages takes into account those who use the templates to indicate that they do not wish to be recruited or would only like to be recruited X times within Y time. Note that this deals with recruitment – the asking of users to participate in a study. Users who are recruited are by no means required to actually participate in the study if they do not want to. -- PiperNigrum (hail|scan) 00:01, 7 March 2010 (UTC)
What do we need a committee for when users can indicate their preferences? Paradoctor (talk) 00:15, 7 March 2010 (UTC)
The reason for SRAG is to be the body that oversees use of the bot. While users can indicate their preferences for contact, the bot shouldn't just sit there and allow anyone to use it to send messages to users. As part of approving use of the bot, researchers indicate what study they are conducting, what IRB approvals they have, and the type and number of users they would like to ask for participation, and then allows for discussion. Prior experience shows that there is a need for the community to have a means of identifying that research is being conducted. -- PiperNigrum (hail|scan) 16:11, 7 March 2010 (UTC)
"bot shouldn't just sit there and allow anyone to use it": So you want a committe overseeing MiszaBot III, too? Besides, automated messaging isn't exactly a difficult task. Give me list of usernames, Notepad++, and one hour for every 500 messages. For larger tasks, writing a little automation script is a nobrainer.
Please don't get me wrong, documentation for researchers and the bot are good ideas that fully deserve support. It's just that you're insisting on a buzzsaw where a kitchen knife is entirely sufficient. You have not demonstrated why a committee is essential to assisting researchers in acquiring participants. Try to see it from this perspective: Even the current, imperfect state of affairs seems tolerable, at least. Paradoctor (talk) 17:11, 7 March 2010 (UTC)
I agree that the bot itself would be simple to implement. However, I think that if someone were to submit a bot that does that to WP:BAG for researchers to use, they would not look favorably on it without some controls to prevent it from being misused for example to, for example, post commercial messages onto everybody's talk page.
Setting aside SRAG's bot responsibilities, the other purpose of SRAG is to give community acceptance to a study that is recruiting for subjects. One of the things researchers encounter on posting a call for participants are a lot of messages along the lines of "who the !@#$% are you", "do you even have IRB approval? I can't believe they'd grant it", "your study design is flawed, since you don't have a control group", etc. messages. Having SRAG as a place to post and discuss studies prior to recruitment would give the researchers a place to point to and say "It's gone through community discussion here." Now, I'm not saying that it will make all of that go away, but it would greatly cut down the number of these kinds of things.
-- PiperNigrum (hail|scan) 19:53, 7 March 2010 (UTC)
"commercial messages": Come on, I already said that spammers don't rely on our bots. And to sort out UCMs, we don't need a committee, edit protection for the bot's commando page is sufficient.
"community acceptance" ... "the things researchers encounter": That's something the researchers can handle themselves, e. g. Village Pump (misc). I don't think an anthropologist will parachute into the middle of the village and start shooting. ;) Paradoctor (talk) 20:18, 7 March 2010 (UTC)
Afterthought: If such a bot is going to be created, why not go the whole 9 mm, and make it a general WP:CanvassBot? Work is the same, and who knows what good use it might be put to? (Delivering Signpost?) Paradoctor (talk) 20:23, 7 March 2010 (UTC)
I'm with Paradoctor on this. we already have rules such as wp:canvass and I don't see the need for a filter where one group claim exclusive rights to vet research projects. I've taken part in a couple of research projects on the pedia and I do see the need for a project that aims to document all such initiatives, collate the findings and can advise researchers and hopefully prevent reinvention of wheels. But no I don't want a bureaucracy claiming exclusive rights to approve projects on the pedia. ϢereSpielChequers 00:17, 7 March 2010 (UTC)
One problem this policy is designed to improve is that researchers often face powerful blowback from users who question whether doing research on Wikipedia is appropriate at all. Having a review process will be important in responding to such users. I do think it's important not to see this as "one group claiming exclusive rights"; if you have experience in research on Wikipedia you should volunteer to help evaluating proposals. As with all of Wikipedia, the community should be making and enforcing the rules. It would be very unhealthy if somehow one group were to take over the execution of this policy. That is why many different researchers are helping to define the policy, and will be helping in its execution. --Riedl (talk) 23:55, 10 March 2010 (UTC)
It should be apparent that the proposed policy does not give SRAG exclusive rights to vet and approve all research projects. The scope of SRAG has been narrowly focused to deal specifically with research that requires contacting a large number of users via talk page postings. Research that does not involve this point need not go to SRAG at all.
As a successfully recruited participant in research, you may not see the difficulty researchers have in requesting participation from users of the community. As indicated in my reply above to Paradoctor, prior experience has shown that something like this is needed. It was suggested that rather than approaching Wikimedia and having something handed down à la WP:OFFICE, it would be more acceptable and wiki-like if we work from the bottom up and incorporate this into Wikipedia policy.
Regarding your other suggestion, WikiProject Research has been created for researchers to come and discuss other issues related to research of Wikipedia. Perhaps you should suggest your idea there.
-- PiperNigrum (hail|scan) 16:11, 7 March 2010 (UTC)

Again

I'm not seeing the problem -- for Wikipedia.

I see the problem for the outside researchers, who didn't get to do their surveys. But how does the current state of things hurt the encyclopedia?

Was article content impaired because an outside researcher's website was blacklisted? Did anything get vandalized when editors complained about spamming notices to dozens of editor pages?

And, frankly, does anyone really believe that an editor who complains about a survey invitation now will magically not complain about an "authorized" survey invitation later? WhatamIdoing (talk) 20:15, 1 April 2010 (UTC)

Wikipedia, as a community, benefits from academic research. From the proposed policy:

In the past, research in Wikipedia has built an understanding of how Wikipedia works,[1] how editors interact with each other,[2] what work is discarded and why,[3] how admins are chosen,[4][5] and how to detect vandalism.[6][7]

The cited research above took place without the need for policy like this one because it is the result of simply analyzing the revision history of the encyclopedia. The proposed policy defines how research that requires interactions with users would take place. This is important to the community since research of this nature is usually directed towards improving productivity, increasing article quality, reducing the effects of vandalism, easing communications between community members or otherwise figuring out how to improve the community as a whole.
Wikipedia's community has smart people from social psychology, computer science and other fields working hard to understand and improve how the system works. This policy is a product of those researchers and other Wikipedians interested in seeing this research continue hashing out the best way to expand the types of research that can be done while keeping control over how these interactions can be initiated in the hands of the community. --EpochFail(talk|work) 15:05, 2 April 2010 (UTC)
Sorry, but I fail to believe that the purpose of outside research is to improve Wikipedia's efficiency or content.
We might occasionally be able to use some information toward that end, but their purpose is to find out what is happening, not to provide advice on improving it. WhatamIdoing (talk) 20:10, 2 April 2010 (UTC)
A lot (maybe most) of the research I have seen published so far has been studying English Wikipedia from a systems perspective, using quantitative approaches to understand "what is happening". There has also been other studies that take a different approach; the UCSC WikiLab created the WikiTrust extension; there's Suggestbot which aims to help editors find articles in need of work; and out of PARC came WikiDashboard.
Because so much data is easily available to researchers makes quantitative systems analysis appear a lot easier to do than research requiring (close) contact with users, so I don't find it surprising that most research is of that kind. As should be evident from the discussion around this policy, experiences with doing intervention research have been somewhat problematic, and the policy aims to mitigate that. Nettrom (talk) 21:28, 2 April 2010 (UTC)
♦ What, the two assumptions are (1) learning how a community works is beneficial for that community and (2) it is useful for a community to have a diversity of techniques in learning how it works. Having scholarly researchers study the encyclopedia and its processes increases that diversity. --R27182818 (talk) 20:13, 3 April 2010 (UTC)

What SRAG is good for, maybe, from a semi-outsider.

I've done WP research before, as the person who for better or worse is the current owner of SuggestBot (though it's in the process of being moved to a more active caretaker). Someone asked me to read this discussion and comment. So:

Is the problem mostly the idea of "controlling who can post"? If the policy talked about establishing a broadly community-recognized mechanism for proposing, coordinating, legitimizing, and supporting both the execution of these kinds of many-posting experiments and tools for individual Wikipedians to opt out of them, would that be a clearer and more acceptable statement?

From what I'm reading one of the big goals is to have an experimental framework both that gives a standard best-practices way of conducting these things, and another is to be able to, when someone says "who the #*(Q%#@ are you?", point to having gone though SRAG as both a way to tell people about the academic research activity on Wikipedia and to be reassuring that some representative sample of Wikipedians has reviewed the experiment and found it to be "okay to do".

Maybe I'm reading this wrong, but that's what I see right now. It doesn't seem like a bad tool to have. I do think that people who want to do things their own way should be able to, but they face the practical issues that people on both sides of the policy have pointed about above, around how people react to the invitation/solicitation.

I'm probably not going to have too much more to contribute here, but this is what I see the discussion (so far) as being about. -- 128.84.103.49 (talk) 20:56, 10 March 2010 (UTC)

I think you have described the purpose of WP:Research and WP:SRAG correctly. Wikipedia should have WP:Research (and, of course, SRAG) as policy because research of Wikipedia is important and should take place, but it is more important that the community is in control of how researchers are allowed operate. Currently, randomly recruiting users via talk page postings is not covered by any policy, and for that reason, confusion and conflict are common. The community currently has no control and researchers do not have a way to go about recruiting users without strong backlash. This proposed policy and SRAG would both offer community control over when researchers get to recruit and give researchers legitimacy when they try to recruit subjects. --EpochFail(talk|work) 22:05, 10 March 2010 (UTC)

Concerns

Now that I've gone against my personal preference and upgraded this to policy, I'd like to suggest a few changes.

  • This section defines the Subject Recruitment Approvals Group to control recruitment message postings.
  • This section defines the Subject Recruitment Approvals Group to assist in message postings.
  • As a rule of thumb, if a researcher wants to contact an individual with the intent of collecting data for use in a study, and that contact is unsolicited, he or she will need approval from WP:SRAG.
  • As a rule of thumb, if a researcher wants to contact an individual with the intent of collecting data for use in a study, and that contact is unsolicited, he or should seek guidance from WP:SRAG.

These turn this adminstrative checklist into the form of wikipedia policy. Hipocrite (talk) 20:41, 7 April 2010 (UTC)

♦ Hipocrite, can you elaborate on your reasoning or give some links? I'm worried that having a policy that is apparently optional will do little to avoid the unpleasant interactions which this policy seeks to avoid. One thing to keep in mind is that an important audience of this policy is researchers who are not familiar with the community. So if SRAG's role is simply to "assist" and researchers "should" comply, then I worry they'll just ignore it.
If it's simply a style issue and the intent is still to have all researchers go through SRAG, then the true meaning of the wording needs to be clarified for people who aren't familiar with WP.
Personally, I believe that all researchers really do need to go through SRAG, because unmediated interactions (which are known to frequently go very badly) will lead to researchers in general, not just those who didn't go through SRAG, being increasingly unwelcome on Wikipedia.
Lastly, I'm concerned about making these changes without a deeper consensus, since they change the policy in a fundamental way. If you don't mind, I'd like to revert them while discussion proceeds. --R27182818 (talk) 17:29, 8 April 2010 (UTC)
I suggest that your ability to require people to do something implies that you have force behind you. As you are aware, policy is descriptive, not proscriptive. While it is possible that failure to go through SRAG would result in blocks or bans, that is currently far from certain. Additionally, making the process a "should" as opposed to a "must" gets you everything you want - a cover for researchers and a process, without at the same time creating a typically community opposed level of beurocracy. Hipocrite (talk) 14:16, 9 April 2010 (UTC)
It seems to me that what is needed is two things: guidance for researchers on how to interact with the community, and a means for researchers to contact Wikipedia users to participate in research without excessive and destructive backlash. For the first of these, a guideline seems to me to be sufficient. For the second case, this seems to be the role WP:SRAG and the forthcoming User:SubjectRecruitmentBot play: a means to contact users, with the ability to opt out of such contacts, in an approved fashion with appropriate controls to prevent abuse.
The difficulty with this is "blessing" SRAG and the bot so that researchers properly executing a SRAG-approved study have a good defense against complaints; ideally, complaints about a properly vetted and executed recruitment should go before the SRAG first or be considered inappropriate. Is anything short of a policy sufficient for this? Is simply getting SubjectRecruitmentBot and the SRAG approved enough? Elehack (talk) 17:23, 9 April 2010 (UTC)
I don't see the strong statements like the ones you've cited as asserting that there is force behind anything. I see them as stating, in no uncertain terms, what action are and are not in line with this "policy". The consequences of breaking the policy are left undefined. --EpochFail(talk|work) 20:11, 9 April 2010 (UTC)

Guideline vs Policy

Despite what may be perceived as critical comments, I don't object to the overall goals here, and I am confident that this page will improve incrementally over time, until it reaches perfection shortly before the WP:DEADLINE.

I am currently curious why {{Policy}} was chosen over {{Guideline}}. Proposals for new policies make some editors irrationally twitchy. Is there some perceived advantage to "policy" status? They're really not that different. WhatamIdoing (talk) 04:17, 8 April 2010 (UTC)

While policy and guideline seem similar, the key difference is a "must" vs "should". We wanted to have it stated as strongly as possible (well, as strongly as is possible on Wikipedia where "must" can somtimes mean "maybe") that subject recruitment happen via SRAG in order to prevent situations like this. Also, from a researcher perspective, it's better to hand them a Policy that they follow ("proscriptive"), rather than a Guideline ("descriptive") that they might disregard because they think it unnecessary. -- PiperNigrum (hail|scan) 15:18, 8 April 2010 (UTC)
I see. May I suggest that you read Wikipedia:The difference between policies, guidelines and essays? WhatamIdoing (talk) 17:41, 9 April 2010 (UTC)
I believe that polices are the underlying "whats" about editing wikipedia, the guidelines are the "hows" and essays were an opinion on "best practices." I picked policy when I evaluated because this page was not only a process on how to conduct research (a guideline), but also an ironclad set of rules about conducting research (a policy). I should note for the record that my personal opinion in evaluating the RFC is that consensus was wrong, and that the page as it stood was patently unaceptable. However, at the time, consensus was clear. It is no longer. Hipocrite (talk) 17:45, 9 April 2010 (UTC)
First, on a side note, would you mind explaining (possibly in another topic on this talk page) why you thought that the page as it stood at the time of the RFC was "patently unacceptable"? I'd like to see if I can do some re-writing to address your concerns.
I think that WP:SRAG is largely about "how" one goes about recruiting editors on Wikipedia, whereas WP:Research is about "who" researchers are, "what" researchers are doing, "what" research is, "what" will be considered "research subject recruitment". WP:Research delegates to WP:SRAG for "how" to go about doing recruitment. WP:Research has strong language about what types of recruitment will be acceptable by the Wikipedia community, but it does not state "how" the community should select those types of recruitment. --EpochFail(talk|work) 20:29, 9 April 2010 (UTC)
You appear aware of my concerns. You created a mandated new burocracy. I don't believe that mandating a new beurocracy is acceptable. Remove the mandatory new burocracy. Hipocrite (talk) 20:43, 9 April 2010 (UTC)
Would it be more reasonable to take the following tack: the SubjectRecruitmentBot, pursuant to approval by the SRAG, is an acceptable way to recruit subjects for research studies. Nothing then directly prohibits a researcher from sidestepping that and attempting other recruitment methods, but then the burden is on them of making sure that they don't violate any relevant policies like WP:CANVAS. The SRAG and bot simply provide an easy path to a minimal-risk recruitment (and the SRAG approval process should work out all policy problems before the recruitment takes place).
The potential problem I see with this approach is poorly-executed, non-SRAG studies causing backlash against the research in general that makes life more difficult for the researchers who are trying hard to work with the community to make legitimate research possible. However, the defense that "these researchers you're complaining about didn't go through SRAG, why are you mad about us?" should be reasonable, and hopefully there are enough cool heads in such situations to allow good research to continue. Elehack (talk) 18:20, 10 April 2010 (UTC)
♦ Two follow-up questions:
  1. If the policy were relaxed from "must" to "should" as you suggest, would you then support it (i.e., is that your only objection or are there more)?
  2. Do you believe that essentially all research which recruits volunteers on Wikipedia should go through SRAG? I.e., do we disagree on wording/style or on something more fundamental?
Thanks! --R27182818 (talk) 18:44, 11 April 2010 (UTC)

Points of non-consensus

♦ With the new interest and controversy brewing, can we record somewhere (and keep up-to-date) what parts of this policy are controversial and which seem to have reached consensus? And also the principal positions for each point of disagreement and the rationale for those positions? I'm confused myself. --R27182818 (talk) 22:03, 11 April 2010 (UTC)

"Rationale" Section needed?

♦ There's been a lot of traffic here and elsewhere wondering why this policy is needed. The answers are buried in this lengthy talk page and its archives (and perhaps other places? I don't recall). Should we add a Rationale section to the policy page itself summarizing the arguments? --R27182818 (talk) 17:29, 8 April 2010 (UTC)

I'd second that... sounds like a wonderful addition to me. Elehack (talk) 20:28, 8 April 2010 (UTC)
Agreed. I'll start by trying to list out the essential points that will need to be covered. Please feel free to edit this space. --EpochFail(talk|work) 21:39, 8 April 2010 (UTC)

Why is this good for Wikipedia?

  • Assumptions
    • It is beneficial for a community to learn about itself.
    • The more approaches to learning about a community, the better.
    • Most Wikipedians are happy to be asked to volunteer for research studies, but some want to be left alone.
  • Assertions
    • Research is one approach to increasing knowledge about the Wikipedia community.
    • This policy allows the community to control when and how this research takes place.
    • This policy enables researchers to perform important types of research within Wikipedia.

Why does it need to be a policy?

  • Without a policy:
    • While researchers try to be respectful, they are often ignorant of community norms.
      • Since researchers are often outsiders to the community, they tend to break policy or otherwise disrupt users without intending to.
      • These breaches of norms tend to be met with vitriol from a very small subset of the community.
      • This reaction from the community has stopped more than one study from happening.
    • Editors who do not want to be bothered by research studies have no structured way to express this until they are recruited.
    • Research that requires interaction with editors is difficult to impossible, so these types of studies don't happen.
  • With a policy, the community can hold researchers to the standards they expect.
    • Researchers can find out, through the recruitment approval process, exactly what is expected of them.
    • With the creation of SRAG, the community has a central place to review how researchers plan to go about their studies before the study begins.
      • This should result in less disruption to editors' work.
      • SRAG and this policy are a constructive outlets for editors who do not approve of the way recruiting is done or the type of studies that are being approved.
    • Editors who wish not to be bothered can opt-out of research recruitment requests via a standardized template. (see User:SubjectRecruitmentBot for details)

Why should the policy be mandatory and not suggested?

  • Having a policy that is apparently optional will lead to non-compliance.
    • i.e., if SRAG's role is simply to "assist" and researchers "should" comply, the concern is that it will be ignored.
  • An important audience is researchers who are not familiar with the community, so they will not understand the nuances of typical policy language.
  • If some researchers keep having unmediated interactions (which are known to often go very badly), that will lead to researchers in general, not just those who didn't go through SRAG, being increasingly unwelcome on Wikipedia.
I might reword the "breeches of policy tend to be met..." point, but other than that I think these reasonings look good. Elehack (talk) 17:26, 9 April 2010 (UTC)
♦ Added points on why it should be mandatory, and correct spelling of "breaches". Otherwise good. I think it's ready for prosification. --R27182818 (talk) 22:03, 11 April 2010 (UTC)
I disagree with most of the "mandatory and not suggested" section. I don't believe that most researchers are stupid, and therefore I don't believe that researchers who have (magically) discovered the existence of the SRAG will be non-compliant.
I think that the lack of understanding about "nuances of typical [for Wikipedia] policy language" are likely to overinterpret, not underinterpret, the requirements.
"Mediation" by SRAG is still going to result in complaints. The research I've seen has generally involved questions that seem silly to editors (but might, I suppose, make sense to a complete outsider). Authorizing talk page spam is not going to make the researchers' questions any smarter.
Finally, both the motivation (that researchers can never WP:IAR) and the solution (we now declare extensive reasons why compliance with our bureaucratic procedure is mandatory) seem unprecedented and incomparable to all other policies. WhatamIdoing (talk) 20:36, 12 April 2010 (UTC)
Really, all that needs to be said (IMO) is that Wikipedia:Do not disrupt Wikipedia to illustrate a point applies to academic research as well. If it's not disruptive, nobody cares. If it disrupts content pages (e.g. typical "vandalize and see what happens" experiments), deny them (and explain why). If it's unsolicited spam, then most of the angry responses you're going to get won't care if it's legitimate research or not, even with an opt-out, even if it was described clearly how to opt out on this page (and it isn't). Nifboy (talk) 02:47, 13 April 2010 (UTC)
That, and we should indeed have a bureaucracy-free point of contact in the nature of WP:SUP. The rest of this is perfectly useless. — Gavia immer (talk) 02:54, 13 April 2010 (UTC)
Wikipedia:Do not disrupt Wikipedia to illustrate a point is insufficient for research related posting as evidenced by the often linked examples of unmediated interactions. Without structure and a way for editors to opt out of recruitment postings, studies often fail before they have a chance and editors resentment towards being researched grows. Even the best recruitment efforts are met with threats of banned accounts and deleted user pages regardless of whether WP:CANVASS can be applied or not.
It seems that many of the criticisms in this thread attack WP:Research because it introduces bureaucracy into Wikipedia. I agree that such processes, if they were to effect editing practices, are destructive to the process of getting work done in Wikipedia. What seems to be missed here is that WP:Research would act as a bureaucratic barrier between editors and researchers who'd like to interact with them. This policy would not effect editing in any way. It would only serve to ensure that researchers who are allowed to recruit have had critical parts of their planned research evaluated (see Requirements for SRAG approval) and that they do not bother editors who have expressed the preference to never be asked to participate in any studies. This process works very well for WP:BAG and is, therefor, not without precedent. --EpochFail(talk|work) 14:33, 13 April 2010 (UTC)

Research discussion

There was an RFC, but it was very short and it is apparent that most of the community has no idea what happened. I demand that the whole community review this before we implement it.

I don't think we're ready for that yet. It's fairly apparent that there is still work to be done on hammering out the exact nature of the recommended policy. I think it would be best to wait a bit and submit a new RFC after the current issues have been discussed more thoroughly by those presently involved. Elehack (talk) 22:21, 11 April 2010 (UTC)
The RFC for this page was open for over 30 days and was advertised in WP:VPP and the mailing lists during that time. It was not short in any way and those who have argued that it was under-advertised have relented when shown the examples of where/when it was advertised. --EpochFail(talk|work) 15:53, 13 April 2010 (UTC)

This is just to collect thoughts from the community and get some air. I think this is very pertinent, we don't do backroom deals, as you know. Air is always good.--Ipatrol (talk) 22:40, 11 April 2010 (UTC)

Well, I didn't know this whole thing existed. Seems like pointless bureaucracy to me. Gigs (talk) 19:18, 12 April 2010 (UTC)

No offense to those who wrote the proposal, but having come here through WP:CENT all I see is an extremely winding puff piece directed at editors attempting to justify the existence of researchers on Wikipedia, appended by a very narrow rule-based (read: WP:CREEP-looking) regulation/permission of contacting editors on their talk pages for research purposes. The former looks entirely too essay-ish to be policy, and the latter looks like an attempt to legislate into business a separate page, the Wikipedia:Subject Recruitment Approvals Group. If there's a point to this page beyond establishing the SRAG, for which this page is not necessary to do, I don't see it. Nifboy (talk) 02:00, 13 April 2010 (UTC)

Research on Wikipedia doesn't require justification. It's happening, and has been for several years. This policy simply makes the community aware of this and that they may be requested to participate unless they do not wish to be. It also defines, for researchers, how to appropriately conduct recruitment on Wikipedia. -- PiperNigrum (hail|scan) 14:55, 13 April 2010 (UTC)
This page does not really "make the community aware of" anything -- not even of the existence of this page. If you want a majority of editors to actually be aware of the existence of research/this page/anything else, then we'll probably have to do some sort of banner announcement. WhatamIdoing (talk) 17:10, 13 April 2010 (UTC)
For an example of exactly the type of policy/guideline these two pages were modeled after, see Wikipedia:Bot_policy and WP:BAG. Could you explain why you see "very narrow, rule-based" regulation as inappropriate in the case of forming a barrier between your talk page and the researchers who might want to recruit you.
As for the point of this page beyond creating SRAG, WP:Research is about what researchers must do while WP:SRAG describes how it will be done. --EpochFail(talk|work) 14:56, 13 April 2010 (UTC)
The bot policy came out of an actual need... bots can be disruptive. Someone soliciting a few random editors for a study isn't disruptive. Where is all the disruption from research that justifies this massive new bureaucracy? Gigs (talk) 15:56, 13 April 2010 (UTC)
Please explain how this bureaucracy is "massive" or otherwise overly large. For examples of disruption, see examples of unmediated interactions.--EpochFail(talk|work) 16:05, 13 April 2010 (UTC)
3 or 4 problems over the last few years (two of which you were involved with), doesn't seem like something meriting a process as heavyweight as designating a special group of people with powers to approve or deny research activities, an application process, and this policy. Gigs (talk) 17:20, 13 April 2010 (UTC)
I was uninvolved in any of those recruitment problems. Even if I were, it doesn't make them any less real. In the research community, we tend to be very hesitant when we see someone else have trouble entering a community successfully. I saw the trouble that Katie Panciera had and decided to try to fix the problem. Also, I suggest you read WP:SRAG. The approvals group is only there to help ensure discussions take place and participate in them themselves. The only power they might have beyond any other editor is control over SubjectRecruitmentBot, but that is more of an obligation than a power since they are subject to consensus. --EpochFail(talk|work) 18:52, 13 April 2010 (UTC)
I've just realized that I was, in fact, involved in one of those unmediated recruitment interactions. My apologies for the mistake. Though, my previous comment still stands. --EpochFail(talk|work) 18:54, 13 April 2010 (UTC)
Agreed. I don't feel the problem has been appropriately identified, and thus the proposal feels like a "solution in search of a problem". It makes researchers jump through more hoops than necessary, and this policy page exists only as an apology, "Sorry we let researchers spam your talk page and dozens of others without telling you". Nifboy (talk) 17:33, 13 April 2010 (UTC)
I find it odd that you think researchers are jumping through too many hoops while the researchers are excited that they finally have hoops to jump through. Also, this policy does not let anyone spam anything. Currently, there is no policy stopping researchers from posting recruitment messages wherever they like (assuming they aren't intending to be disruptive). Instead this policy limits those postings. --EpochFail(talk|work) 18:57, 13 April 2010 (UTC)
I'd cite WP:DICK if somebody started posting messages on user talk pages en masse. That's what I don't understand, is that, if there isn't a problem with such messaging, why a public vetting a la BAG is necessary, and if there is a problem, why we are explicitly allowing an exception so that researchers can be disruptive. Nifboy (talk) 19:32, 13 April 2010 (UTC)
There isn't a problem with this kind of messaging except to those who believe that it is a problem because it's not explicitly allowed. There's two schools of thought here: "anything not explicitly denied is allowed", and "anything not explicitly allowed is denied." Recruitment requests fall within the rules on Wikipedia regarding mass posting, as they are limited, small scale, targeted messages. However, researchers often run into a small but vocal group who seem to follow the second philosophy. -- PiperNigrum (hail|scan) 20:09, 13 April 2010 (UTC)

I saw that list of posts and I was, quite frankly, underwhelmed. Something as major as this deserves a place where it can stand out. There's just so many dmn proposals going around that this drowned in a sea of wikitext. Most users who commented seemed to have stumbled across it by accident. I would like to see this on MediaWiki:Watchlist-details and actually publicized.--Ipatrol (talk) 22:28, 13 April 2010 (UTC)

Yeah, this is something that potentially effects all editors, because of its opt-out nature. That's why I'm so wary of it. Nifboy (talk) 22:48, 13 April 2010 (UTC)
The opt-out mechanism is simply an extension of the current state of affairs. Today, if someone posts a recruitment message on your talk page, you can tell them that you don't want to participate in further research, but that doesn't stop a different researcher from recruiting you afterward. With this mechanism, you can use the template to indicate to all recruitment that you are not interested (or even, that you do not want to be recruited for any research for some fixed period of time). The process is made better for you with this mechanism than without it. -- PiperNigrum (hail|scan) 23:21, 13 April 2010 (UTC)
I don't believe consensus has been established whether canvassing by researchers is even appropriate. You assume that it is, but I'm very uncomfortable making that same assumption, especially when the trouble researchers have had in the past has been with people not believing it's appropriate. Nifboy (talk) 01:15, 14 April 2010 (UTC)
If you read the discussions in the linked to KatherinePanciera example (it's the one I am most familiar with), you'll see that it is a very small subset of those involved had a problem with recruitment messages being posted. Also, if you'll examine the discussion on the notice board, you'll see that there was really only one editor who was upset. You might also notice that many editors suggested contacting the foundation for help. The idea for WP:Research was born of a discussions with representatives from the foundation at WikiSym 2009. --EpochFail(talk|work) 02:12, 14 April 2010 (UTC)
Quoting one of the participants on ANI, "correct way to canvass, posting on wikiprojects and related topic pages". Everyone else there seemed to be under the impression a 50-user canvassing was not to be repeated. Nifboy (talk) 02:32, 14 April 2010 (UTC)
The difficulty with an opt-in mechanism is that people need to be invited to opt-in - effectively that would be defacto canvassing anyway. Josh Parris 03:57, 14 April 2010 (UTC)
Opt-in would be a nightmare. I cannot imagine spamming announcements to all forty thousand of en.wikipedia.org's active editors to alert them that someday, a tiny fraction of them might receive a single talk-page note. WhatamIdoing (talk) 21:27, 14 April 2010 (UTC)
That's exactly what MediaWiki:Watchlist-details is for; communication to all currently active editors. Nifboy (talk) 21:38, 14 April 2010 (UTC)
This proposal needs to be widely advertised either way, so that people who don't want to participate don't get blindsided by the fact that we seem to have now legitimized canvassing, albeit only for legitimate research; the usual way of doing this is through MediaWiki:Watchlist-details. Nifboy (talk) 04:40, 14 April 2010 (UTC)
Blindsided? By a single message posted to their talk page by a bot? I'm not sure this warrants that much concern. --EpochFail(talk|work) 16:57, 14 April 2010 (UTC)
You mean the same kind of canvassing that spawns ANI threads like the one you pointed me towards? Yes. Nifboy (talk) 17:32, 14 April 2010 (UTC)
Spawned by a single individual who wasn't even a recipient of the message in question, who could have then been pointed at something like this policy had it been in existance at that time. -- PiperNigrum (hail|scan) 18:09, 14 April 2010 (UTC)
If you read through that thread, you'd notice that most of the participants thought the postings were not a problem and there was no need to bring it to ANI. The editor who initialized the conversation was reminded by a few people to relax a little and even accused of being the truly disruptive element. He also makes it apparent that he has no idea who researchers are or what they are doing on Wikipedia. --EpochFail(talk|work) 18:57, 14 April 2010 (UTC)
  1. ^ Kittur, Aniket (2007). Power of the few vs. wisdom of the crowd: Wikipedia and the rise of the bourgeoisie (PDF). alt.CHI at Conference on Human Factors in Computing Systems. Retrieved 2009-12-29. {{cite conference}}: External link in |conferenceurl= (help); Unknown parameter |coauthors= ignored (|author= suggested) (help); Unknown parameter |conferenceurl= ignored (|conference-url= suggested) (help)
  2. ^ Kittur, Aniket (2007). "He Says, She Says: Conflict and Coordination in Wikipedia" (PDF). Proceedings of the SIGCHI conference on Human factors in computing systems. Conference on Human Factors in Computing Systems. Vol. 1. ACM Press. pp. 453–462. doi:10.1145/1240624.1240698. ISBN 9781595935939. Retrieved 2009-12-29. {{cite conference}}: External link in |conferenceurl= (help); Unknown parameter |booktitle= ignored (|book-title= suggested) (help); Unknown parameter |coauthors= ignored (|author= suggested) (help); Unknown parameter |conferenceurl= ignored (|conference-url= suggested) (help)
  3. ^ Halfaker, Aaron (2009). "A jury of your peers: Quality experience and Ownership in Wikipedia". Proceedings of the 5th International Symposium on Wikis and Open Collaboration. International Symposium on Wikis. ACM Press. doi:10.1145/1641309.1641332. ISBN 9781605587301. Retrieved 2009-12-29. {{cite conference}}: External link in |conferenceurl= (help); Unknown parameter |booktitle= ignored (|book-title= suggested) (help); Unknown parameter |coauthors= ignored (|author= suggested) (help); Unknown parameter |conferenceurl= ignored (|conference-url= suggested) (help)
  4. ^ Burke, Moira (2008). "Taking Up the Mop: Identifying Future Wikipedia Administrators" (PDF). CHI '08 extended abstracts on Human factors in computing systems. Conference on Human Factors in Computing Systems. ACM Press. pp. 3441–3446. doi:10.1145/1358628.1358871. ISBN 978160558012X. Retrieved 2009-12-29. {{cite conference}}: Check |isbn= value: invalid character (help); External link in |conferenceurl= (help); Unknown parameter |booktitle= ignored (|book-title= suggested) (help); Unknown parameter |coauthors= ignored (|author= suggested) (help); Unknown parameter |conferenceurl= ignored (|conference-url= suggested) (help)
  5. ^ Panciera, Katherine (2009). "Wikipedians Are Born, Not Made". Proceedings of the ACM 2009 international conference on Supporting group work. Conference on Supporting Group Work. ACM Press. pp. 51–60. doi:10.1145/1531674.1531682. ISBN 9781605585000. Retrieved 2009-12-29. {{cite conference}}: External link in |conferenceurl= (help); Unknown parameter |booktitle= ignored (|book-title= suggested) (help); Unknown parameter |coauthors= ignored (|author= suggested) (help); Unknown parameter |conferenceurl= ignored (|conference-url= suggested) (help)
  6. ^ Viégas, Fernanda (2004). "Studying Cooperation and Conflict between Authors with history flow Visualizations" (PDF). Proceedings of the SIGCHI conference on Human factors in computing systems. Conference on Human Factors in Computing Systems. ACM Press. pp. 575–582. doi:10.1145/985692.985765. ISBN 1581137028. Retrieved 2009-12-29. {{cite conference}}: External link in |conferenceurl= (help); Unknown parameter |booktitle= ignored (|book-title= suggested) (help); Unknown parameter |coauthors= ignored (|author= suggested) (help); Unknown parameter |conferenceurl= ignored (|conference-url= suggested) (help)
  7. ^ Priedhorsky, Reid (2007). "Creating, Destroying, and Restoring Value in Wikipedia". Proceedings of the 2007 international ACM conference on Supporting group work. Conference on Supporting Group Work. ACM Press. pp. 259–268. doi:10.1145/1316624.1316663. ISBN 9781595938459. Retrieved 2009-12-29. {{cite conference}}: External link in |conferenceurl= (help); Unknown parameter |booktitle= ignored (|book-title= suggested) (help); Unknown parameter |coauthors= ignored (|author= suggested) (help); Unknown parameter |conferenceurl= ignored (|conference-url= suggested) (help)
Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia_talk:Research/Archive_1&oldid=1066176276"