User:Kennyle78/sandbox

**Welcome to ROC Party II!**

Diagnostic evaluation

Clinic name	General Consultation	Assessment	Diagnosis	Treatment
Findley Clinic	Y	Y	Y

We are using this page to archive the planning emails for the Second ROC Party, running from January to May 2015 with monthly large group, "umbrella" calls. Planning emails will get copied and pasted here, newest on top. Because this is a Wiki, you can make edits to suggest clarifications and add details or reorganize. Please make improvements. :-)

The correspondence from the first ROC Party (May-June 2014) is now in a separate Wiki page -- "ROC Party I".

==========================================================

• • January 18, 2015**

• • ROC Party II– Afterparty #1**

Hi, all,

Thanks for joining the call for ROC Party II! This is shaping up to be a big group and a varied, intellectually rich set of projects. We are enthusiastic!

This is a long email. It includes a lot of details. Again, this may be most useful if you print it out and have a hard copy for reference. This email is also reproduced in the https://OSF.IO site, under the Wiki tab, on a page for the instructions for ROC Party II. Putting this on OSF guarantees that we always can find it again, versus getting lost in email. Making it a Wiki also means that everyone can make suggestions for improving the instructions.

• • OVERVIEW**

The bulk of the work happens *in between* our monthly calls, and there are several things that should happen in the coming weeks to get the most out of the party. Following these steps makes it more likely than not that you’ll have both a deeper understanding of the concepts, and also a manuscript with at least the Methods, Results, tables and figures ready to go by mid-May.

• • SUGGESTED MENU OF ACTIVITIES for Month #1**

Here’s a suggested “set menu” of activities and a proposed order. This is intended to provide structure, but also be flexible. Like at a good restaurant, the set menu is designed to be enjoyable and successful, but if you know that you want a different order or entrée, then by all means customize!

• • • Week 1.1***

• Goal for the week* – conceptual grounding in ROC and how it is different from t-test and other traditional statistical methods in the social science.

• Logistics:* Make sure you are registered on OSF and Peerwise (goldman3@live.unc.edu is the point person)

• Activities:* Read Youngstrom (2014) “Ready to ROC” primer from J Pediatric Psych (zip file under Templates in OSF). Post at least one multiple choice question on Peerwise, and answer some of the other questions.

• • • Week 1.2***

• Goal for the week:* Start thinking about what dataset and question you would like to be the focus for your subgroup

• Logistics:* Post open-ended questions and logistical questions to the “Questions” Wiki Page in OSF. EAY and others will read and answer a couple times a week.

• Activities:* Play with the sample data from the Youngstrom (2014) primer. The data are included in two formats (SPSS and Excel) in the zip file on OSF, and you also have the syntax, so you can re-run the syntax and see the raw output. This will give you a feel for the methods, and also the types of variables and how to check them for the (few) assumptions involved. Nominate a dataset backchannel and discuss with any collaborators in your own group.

• • • Week 1.3***

• Goal for the week:* Team building; finalize project selection

• Logistics:* We want to have one experienced ROCer, one content expert, one person familiar with the dataset, and one understudy in each project group. There could be overlap in roles. Minimum suggested group size: 3 people. Maximum: ?? (More than 5 probably hits diminishing returns). We will use email to pair up people and projects. Need to start working through any admin issues for data sharing (deidentifying data set, local data use agreement, any other permissions).

• Activities:* Complete ROC Planning Canvas if not already done. https://docs.google.com/drawings/d/1Ujx34TogsFHdehqnjiaCOimhV0oVGdm9Q9CErKiE3SQ/edit

Get feedback from experienced ROCer. Finalize dataset choice and team.

• • • Week 1.4***

• Goal for the week:* Be ready to ROC with new data – project team has met; all of team has data

• Logistics:* complete at least one meeting to “meet and greet” team members; make sure everyone has data and appropriate permissions

• Activities:* Project team meeting; report in to large group on umbrella call #2 (Friday, 2/13, 1:30-2:30 pm EST).

---

• • The ROC Canvas**

https://docs.google.com/drawings/d/1Ujx34TogsFHdehqnjiaCOimhV0oVGdm9Q9CErKiE3SQ/edit This is a one page worksheet designed to help plan the analysis and evaluate whether a dataset will be appropriate. It is a GoogleDoc. If people have trouble accessing it, we can look at converting to a different format.

One of the subtle but powerful things we’ve seen with using the ROC canvas is that it organizes the variables so that the statistical expert and content expert can work together much more quickly. Having someone who knows the dataset specify the exact variable names, and put them in the appropriate categories, reduces or eliminates the time needed for the stats person to learn the dataset. With a good canvas, the team can be running analyses within 5 minutes, instead of it taking hours or days for people to get familiar with the dataset. The minimum to be able to get started with analyses would be to work through the “Design Triage,” “Criterion Definitions” (referring to the diagnosis), and “Predictor Variables” (what we are using to try to classify cases into the correct diagnosis). Distractor and Moderator variables are optional, but good to think about. We will be talking more about those concepts in month 2, and revisiting them then. Clinical Application, Cost Structure, and “TBA” are concepts we’ll delve into later, too.

• • Big Call Logistics:**

It would be great to have a web solution that would let people have video or desktop sharing, and ideally record the meeting. UNC does *not* appear to have a license for GoToMeeting that we can use. Googlehangouts works well for the small groups, in our experience, but does not scale well to the big call. Gerry offered to have Children’s National host the next call on GoToMeeting on their license. If that offer holds, then we will do a test call in the next 2 weeks to beta-test and see if this will work for the 2/13 call.

• • Any other Questions or Suggestions?**

Eric’s email is eay@unc.edu, and cell is 216-410-7975. Using the Wiki on OSF.IO also has the power of drawing on the whole group, so it may be faster. :-)

All the best,

Eric & Team

=========================================================

• • January 15, 2015**

• • ROC Party II -- The Beginning (again)**

Here’s the Join.Me ID: 391-453-639 This is optional, but will let you see my desktop.

Hi, all,

Here are the details for the call tomorrow, and some orientation to the ideas and technology we’ll be using over the course of the semester. This is a long email, doubling as a syllabus of sorts, and an orientation. You’ll want to print this out before the call and have it handy.

Call in information: Phone: 832-551-5100, Conference ID: 202921

Video options: We have tried a lot of video options, with mixed results. Skype crashes my computer if it and SPSS are open at the same time (they are both pigs, and the processor overheats). Join.Me works well. Google Hangouts may not accommodate the full group on one hangout, and people need a Gmail account to link. I will host a Join.Me video link (id will come tomorrow morning. Once we have a list with peoples’ Gmails, we can set up a hangout and experiment with that; trying to build a hangout during the call tomorrow would take a huge chunk of the call time.

We have a big group and don’t all know each other. That’s okay – part of the fun of this will be the networking! 

Agenda for Call 1:20 – call line and Join.Me connection will be open, 1:30 – Start call – welcome and 30-60 second group introductions (UNC, Children’s National, Virginia Tech, Lancaster, etc.) 1:35 – Overview of ROC Party format for Spring (10 min + 5 Q & A) 1:45 – Logistics: Access to OSF, Access to Peerwise, contact information (15 min) 2:00 – First draft of small groups and projects

               Teams typically include at least 3 people – Content Expert, ROC leader, ROC understudy; could have additional students or co-authors

2:20 – Plan for mop-up and next actions 2:30 – Adjourn

That’s a lot to pack into an hour. One of the key concepts is the idea of a flipped classroom – most of the discussion and progress is going to happen in between these calls. The calls are mostly logistics: a report back & organize for the next push. The first ROC party met 4 times, with lots of sidebars and subgroup meetings in between.

The power of the format comes from the combination of a larger timetable, regular goals, and the sharing of resources online. We experimented with a lot of formats and technologies the first time. We have settled on two platforms as core:

• • Open Science Framework (OSF)**

https://osf.io

OSF is where we are sharing a bunch of the templates, source code, and project related materials. If you already have an account, we can link you to the ROC Party project in it. You will need a membership to access all the materials (it is *very* free – you will see that it is designed for us to be able to push things out to the general public on the Web if we wanted).

Peerwise is a teaching tool. It is pretty easy to learn, and we’ll have the advantage of walking into a “class” where the first wave of ROCers already created content. Details about joining below.

We have some things that we have done with Google, but not everyone has a Google address; we can decide backchannel whether we want to use it, too (for Hangouts, or other purposes).

• • Syllabus/Plan for the Semester **

(note that there is a “pre-meeting” set of goals before each group meeting)

We anticipate 4-8 projects for this second run. Let us know if you have any thoughts or suggestions, and definitely let us know if you have a project that you would like to run through the party mechanism. It would be a lot of fun to have you guys involved and sharing in shaping the experience!

• • First Call: Jan 16, 1:30-2:30 pm EST**

Before Next Meeting: Goals: Exposure to ROC concepts and output. Identify viable new dataset for team to work up. All participants read ready to ROC, write a multiple choice question for PEERWISE, answer other questions on PEERWISE; write any discussion questions and circulate 48 hours ahead Facilitators vet dataset through ROC Canvas, discuss feasibility, finalize teams.

• • Second Call: Fri Feb 13, 1:30-2:30 pm **

(30 min afterparty for the facilitators) – Agenda: Discuss questions from Ready to ROC; meet your dataset

Before meeting: try running analyses on variables, make list of questions and save syntax

• • Third Call: Fri March 20 1:30-2:30 pm**

(30 min afterparty for the facilitators) – Report on initial findings, small group discussion, report back on next steps & questions

Before meeting: MONS model – draft results section & tables; circulate drafts 24 hours in advance

• • Fourth Call: April 17, 1:30-2:30 pm**

(30 min afterparty for the facilitators) – small group: discuss drafts, select and integrate; report back

Before meeting: MONS model – draft methods, discussion

• • Fifth Call: May 15, 1:30pm **

(30 min afterparty for the facilitators) – small group: DISCUSS target journals; review drafts, integrate, complete and review AARFS, discuss any other issues prior to submission; review process and make suggestions for the fall ROC party After meeting: finalize drafts

---

• • Peerwise**

https://peerwise.cs.auckland.ac.nz/at/?unc_edu

Course: MECCAPsych

Course ID: 9159

(You make up the identifier and email it to goldman3@live.unc.edu; we have to add it to a "white list" and then you will be able to join the class. Please allow 24 hours for us to get you added).

Peerwise is an online resource for you to create and share multiple choice questions. We are experimenting with this as a way to build up a library of questions about bipolar disorder, assessment, and other topics of interest to use to help people learn the material. To create your account, please send Emily Goldman (goldman3@live.unc.edu) the four digit ID you would like to use (I suggest the last four of your SSN or PID, or something else that you can remember easily). We add the ID to an “approve list” for the class, so please email the four digit ID you would like to use. The Course ID is 9159.

• • Expectations: **

Participation is completely voluntary, and we are aiming for the sweet spot that maximizes fun and productivity. We also are coming to play seriously. This is an assertive timetable, but the rewards are big. For people that commit to being faculty, the bump in salary is estimated at $1000/year for early publications. Over the course of your lives and careers, that compounds into significant moola. For people that choose other paths, the standard deviation is bigger, but the slope is always positive. That is Stat Speak for “Do this, and only good things will happen.”

If you sign up, you are agreeing to:

• Make your best effort to participate in all meetings and activities

• Arrive early or on time

• Come prepared!

• You can opt out of any portion, but please do the courtesy of emailing me and your team ahead of time so that we can adjust accordingly

• • Products: **

Everyone who participates in the “ROC Party” will gain a deeper understanding of ROC, diagnostic efficiency, and clinical decision-making. We also will hone our chops at making tables, figures, and presenting results. The schedule is designed to have a high probability of resulting in a publishable manuscript, but that is partly contingent on the data cooperating. Even if they don’t, the product should still be sufficient to turn into a presentation at a conference, but the data sets and measures we are picking offer as close to a “sure bet” as occurs in the real world.

• The “MONS Model”* is one where everyone on the team individually writes all sections of a project, then the group convenes, compares the drafts, selects one as “primary” and draws in the key modifications from the others. We have done this with posters and found it to work better than carving up pieces for separate authors; this is our first attempt at scaling up to a manuscript level. In terms of authorship, we will use the AARF approach (there is a template file in OSF), with the majority of points for a particular section going to the person whose draft was adopted as the primary version, and remaining points going to others. There will be generous “wingman” points for replication of analyses, given the training goals of the project.

• • “Devil Takes the Laggard” Clause: **

If you don’t have a draft submitted by the deadline, your version of that module doesn’t get considered for inclusion in the final version (unless there are no submitted versions). This is a draconian rule, and we can decide whether or not we want to keep it in the future. This is our time to beta-test it. The intent is to reward a focus on commitments, and to ensure that projects reach completion quickly.

• • • Lawyer zone: ***

Signing up to participate is not enough to get authorship; you have to earn that by participating actively and ideally by improving the product. Attempts to game the system will be subject to loss of letter of recommendation, ridicule, both, or worse. Files and content in OSF that are not public should not be made public without the prior consent of the author or owner of the content.

• • Next Actions: **

1) Email Emily Goldman (goldman3@live.unc.edu) your contact information (Name, Preferred email, four digit code for Peerwise, cell & gmail if you want to share, OSF account name if set up)

2) Register for OSF and Peerwise

3) Facilitators: Nominate your project of choice and make sure that the data are accessible

4) Participants: Make sure to have read the “Ready to ROC” paper, pulled down the sample data, and run the sample analyses in SPSS and/or R. Email a PDF of the output as your ticket to enter the party. If you do it in R, then you’ll accrue bonus points, and probably get to skip having to come back as an opossum sometime in the next dozen reincarnations.

5) Technical: Please make sure that you have a good internet connection, a camera (or a good recent picture to send as a surrogate), and a good phone or mic connection. Once we have divvied up into teams, each team is free to decide what works best for them (some options include Google Hangouts, Join.Me, TeamViewer, Skype, etc.). That is another part of the “beta testing” this time around.

Please email me ASAP or call (216-410-7975) if you have questions or suggestions.

I think that this is going to kick butt, constructively. Looking forward to tearing it up together!

All the best,

Eric & the ROCing Facilitators (sounds like the name of a band! Serendipity?)

=========================================================

• • January 14, 2015**

• • Kickoff email**

Hi, all,

Thanks for your continued interest in the ROC Party!

There’s been a lot percolating in the background.

We have two formats that we are planning: One is a “spring semester” format where we have four large group meetings, once a month (Friday 1:30 to 2:30 EST, starting this week) to make introductions, discuss key points. Most of the work will happen in between meetings, when small groups focused on a specific project would meet and move analyses and drafts along.

The other will be an intensive “short course” format. We are trying to decide whether to do a two day “workshop” format, a week long (which would be a bigger time commitment, but would allow more downtime) or a 5 week format (most similar to what we did with ROC Party I, last May-June).

The initial “interest meeting” and planning call will be this Friday 1:30 to 2:30 EST. If you are interested but can’t make the call, let’s talk backchannel about best options. We have a lot of interest here at UNC, with people who can be “extenders” to help with analyses and logistical support.

We ran 6 projects through the first ROC party, and that seemed like a good number. I think that 4-8 would definitely be in the sweet spot to get the benefits of scale and seeing multiple sets of analyses and technical issues, but not be cumbersome. We have enough projects and data to exceed that capacity, so we definitely are interested in doing both a “one semester” and a summer intensive format. With batches, too, there is the possibility of feeding these into a special section proposal for a journal, and there is interest in an edited volume anthologization. If people are interested in those, we can explore further with publishers (Guilford and Oxford editors have both expressed interest in the concept).

The advantages of the large group component are having some structure and social support, as well as a set of shared resources.

[ ] Sorry, I will miss the call, but I am still interested.

Looking forward to it, and best to all,

Eric