Wikipedia:The Wikipedia Adventure/Impact

Intro

There is interest in how principles of game mechanics and playful design can help encourage users to take meaningful actions online. It is not yet known if that body of research offers lessons for improving Wikipedia, however. As such, we have designed an experiment to test whether an onboarding game is a useful method for training new Wikipedians.

Research questions

Do new editors who complete the Wikipedia Adventure:

• go on to be more active and successful Wikipedians?

• make more positive contributions to the encyclopedia?

• have a better understanding of Wikipedia and experience fewer frustrations?

• remain with the community for longer than editors who are not exposed to it?

Test plan

Alpha phase

An alpha test of the game will be deployed on English Wikipedia in August through October 2013. Bug fixing, problem solving, and basic usability is the key focus of this phase.

• Existing editors will be invited via talk page messages etc to try the game and give feedback.

Beta phase

Following iterations based on the alpha, a beta test will be deployed in November 2013. Impact on new editors is the key focus of this phase.

• A random sample of new editors will be invited (via talk message or banner) to play the game.

• Metrics will be run on the test group and compared with a similar control group not playing the game in order to determine whether the game has any impact on editing activity.

• All editors who start the game will be monitored closely for signs of increased vandalism to the encyclopedia, and cleanup actions will be taken by those monitoring as needed during the test.

Analysis

• FeedtoSnuggle ratings, compare to start point snuggle score, and compare to control group before and after
• Event logging (in guided tours)
• Comparisons: Not invited/Invited but didn't complete mission 1/Completed Mission 1/Completed Mission 4/Completed Mission 7

Mission-specific skills

• • userpage edits
  • user talk edits
  • article space edits
  • ???
  • teahouse edits
  • inline citations added
  • wikilinks, images, headers added

Metrics

Quantitative

We'll run database queries to gather quantitative data including:

• number of edits to articles
• number of talk page edits
• frequency of edits over time
• amount of content added that survives over time
• warnings and blocks

Cohorts

• Editors who made 1 edit and received a TWA invite (but went on to make at least one edit later)
• Editors who made 1 edit and did not receive a TWA invite (but went on to make at least one edit later)

Analysis

• Editors who did not complete mission 1
• Editors who completed TWA mission 1
• Editors who completed TWA mission 4
• Editors who completed TWA mission 7

• Event logging for all steps

Quality

• Number of warnings/blocks
• STiki score - metadata analysis
• Snuggle human classification - on post TWA subset of contribs
• Namespace breakdown

Qualitative

• A survey linked from the last page of the game will be deployed to help assess the editors' satisfaction, experience, and understanding of Wikipedia

Then what?

The results of the experiment will be shared by the end of 2013 in early 2014 with the community in order to inform a decision about whether the game should be more widely deployed, discontinued, or further testing is needed.