Wikipedia:Bots/Requests for approval/HostBot 5
- The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was
Approved.
Contents
Operator: Jtmorgan (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)
Time filed: 14:32, 31 January 2014 (UTC)
Automatic, Supervised, or Manual: Automatic
Programming language(s): Python, uses wikitools
Source code available: WIP, in this repo.
Function overview: Invites new good faith editors to play The Wikipedia Adventure. The Wikipedia Adventure is an interactive onboarding game designed to teach new contributors how to edit in a playful and educational fashion. It's funded by a Wikimedia Foundation Individual Engagement Grant and created by User:Ocaasi. The game takes place entirely in the editor's userspace and no outside Wikipedia pages are touched.
Links to relevant discussions (where appropriate): WP:The Wikipedia Adventure/Impact has the research and impact analysis plan and details, WP:Bots/Requests_for_approval/HostBot_4 has the original invite bot request.
Edit period(s): Daily
Estimated number of pages affected: 300-500 per day depending on the number of quality candidates from Snuggle
Exclusion compliant (Yes/No): Yes
Already has a bot flag (Yes/No): Yes
- Phase 1 impact report
- TWA players made more edits: New editors who played TWA made 1.2x more edits than a control group of similar but non-invited new editors. Players made 1.9x more edits than those who were invited but did not play the game.
- TWA players were more likely to make 20+ edits: TWA players were more likely (1.2-1.7x) to make 20+ edits than either control group. TWA players were also more likely to make 0 edits than the control groups, however.
- Players who finished the game made the most edits: Players who completed the game made 3.2x more edits than those who only started the first level of the game, and were 2.9x more likely to make 20+ edits.
- Players enjoyed the experience and felt more confident: 87% of players surveyed were satisfied or very satisfied overall with the game. 89% said 'TWA made me more confident as an editor.' TWA player: "It really left me feeling prepared to make future edits." 89% said, 'Lots of new editors should be invited to play TWA.'
A possible explanation for these findings: new editors who play TWA appear to be making their test edits within the game rather than on articles. Those who continue to edit Wikipedia after playing demonstrate the confidence of a more highly active editor.
Overall satisfaction
edit- 87% were satisfied or very satisfied overall
- 89% said 'TWA made me more confident as an editor',
- 89% said 'TWA helped me understand Wikipedia better'
- 77% said 'TWA made me want to edit more', 6% disagreed
- 79% said 'TWA made me feel welcomed and supported'
- 71% said, 'TWA helped me know what to do next', 9% disagreed
- 80% said, 'TWA prepared me to be a successful contributor to Wikipedia'
- 75% said, 'I enjoyed playing it', 6% disagreed
- 89% said, 'The game is a good way to introduce new editors to Wikipedia'
- 89% said, 'Lots of new editors should be invited to play The Wikipedia Adventure'
![Overall, how satisfied were you with The Wikipedia Adventure?](http://upload.wikimedia.org/wikipedia/commons/thumb/6/69/TWA_survey_overall.png/700px-TWA_survey_overall.png)
Quote highlights
edit- "I enjoyed the idea of editing a fake article for practice - in fact, when I first saw the game, I immediately hoped it would incorporate some sort of actual editing rather than just theory or questions or something."
- "Well, what's there not to like, or to have an opinion on...the game is great, most-of-all for us users that are just starting up in Wikipedia."
- "I didn't know there was talk and discussion among users until I played the game...I just thought you could make comments and report on individual pages."
- "I've seen and heard companies, including my own, talk about learning through 'gamification'. I found TWA to be the best example of gamification I have witnessed to date."
- "TWA was very informative and helped pull back the curtain on some of the fundamentals of editing."
- "I think TWA at the moment is a great stepping stone for new users such as myself. I would love to see it expand to include more 'advanced' topics that can be optionally covered by the user."
Extended Quantitative and Qualitative analysis
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Quantitative analysiseditWe analyzed 3 groups of 165 editors each in November/December 2013. All were selected from Snuggle's desirability algorithm as likely good-faith contributors. 10,000 editors were invited to play using a mass talk page message invitation.
All groups made 1 edit before they were sampled, and made at least 1 edit afterwards, to ensure that they didn't just wander away. In other words, groups 2 actually saw the invites, and group 3 made an edit after the game as well. We found that:
A note on the analysis: we ran this data after a very brief amount of time--barely 4 weeks. A fuller timeframe for analysis which we will conduct by Spring 2014 will be more robust and have more meaningful signals in them. (We plan to conduct statistical significance tests and reports about editor retention and edit persistence).
Total average editsedit
Article space editsedit
Talk page editsedit
Number of articlesedit
Qualitative analysiseditWe surveyed the 600 editors who at least made it to the first stage of mission 1. We sent these editors talk page invitations to a Qualtrics survey using EdwardsBot. 42 editors responded between December 23rd and January 4th.
![]()
Educational effectivenessedit
![]()
![]()
Design satisfactionedit
The hypothesis we set out to test was that play could be thoughtful and fun could yield meaningful experience and education. The survey data supports this conclusion. We also aimed for a target demographic of college-aged men and women. The most common given age group for appropriateness was that demographic, so it looks like we aimed right. It's also worth noting that the bell curve was fairly 'thick' around this demographic, and survey respondents thoughts TWA would be appropriate for many age ranges, especially those 13-29 (but also younger those than 13 and 55+).
![]()
![]()
![]()
![]()
Player demographicsedit
![]()
![]()
Possibilities for expansionedit
![]()
In their own wordseditWhat they likededit
"The interactiveness of The Wikipedia Adventure was an easier and better way to learn the basics of Wikipedia versus trying to run around to different pages and just reading about it." "Well done basic introduction to Wikipedia." "Simple, easy to use." "It's simple for new Wikipedia editors like me trying to learn the basics of Wikipedia." "Informative and fun" "The conversational tone is pretty good, It makes it fun even if it's all pretty simple." "Gave a very good, brief introduction to editing." "It made stuff easy to understand." "Fun and intuitive game." "I completed the entire game because it wasn't as dry as other training tools out there." "Really enjoyed the entire editing process. A lot of thought was given to it, including the fictional users who guide you through the process. It all flowed very well and was highly educational. I also thought that awarding badges was a nice touch too." "It was all beautifully designed. I enjoyed aspects such as the challenges and badges that made it feel more like an educational tool or game rather than a lecture, and recorded your achievement to date." "It was a nice length and about right level of seriousness for me." "I liked it, very different." "Designed great, easy to use because of that." "I think the Adventure should be kept just as playful. It's definitely not serious, but that's not a bad thing. Maybe it might be off-putting for someone who thinks it's too "cheesy" or doesn't like its tone, but I certainly enjoyed it and I'd bet a lot of other people would, too. "It really left me feeling prepared to make future edits."
What they didn't likeedit
"A little too silly for my liking, but it's probably great for young editors." "Too long-winded and geeky" "The badges are kinda neutral but the whole thing works very well." "The forced badges I had to edit out of my talk page were a bit annoying." "I disliked the way that participants are addressed, as if imbeciles." "I dislike it because it's like a kid's comic book - first impressions are everything - and I did not like it from the point the big alieny picture arrived." "I wouldn't disagree with maybe a separate, more formal introductory "page"." "Maybe split into young-adult and adult streams?" "Personal I think if you replace the space unicorn with a crashed rocket it would be good for anyone with a sense of fun." "Even though it felt somewhat silly playing game, it was a good learning experience. It should be serious and have more techniques and points for serious editors." "Where's the 0-2 category - that is the appropriate level for this stuff - it's at the level of Teletubbies" "Maybe age specific versions? Space for kids, a fictional vampire wiki for teens, university class stuff for students, etc, etc? "I still don't know what the blue guide creature is." "I wish it gave the impression that editors were expected to be mature and intelligent, rather than idiots who could be entertained an educated with this kind of drivel."
I'd like to note that the most negative feedback consistently came from one respondent who had 100,000+ edits. While I do not discount their points--echoed by earlier design debates about the game's playful or even youthful nature--it needs repeating that the target for the game is new editors, and these contributors are different and have different needs than experienced contributors.
What they wanted more ofedit
"May be worth including some information about the policy regarding not editing on behalf of an organisation you belong to. As well as a some additional missions covering what notable enough to be included in Wikipedia." "Maybe it could extend to more complex rules - when I signed up, I found surprisingly few links to policies or guidelines. For instance, one thing that could be included would be something about red links: I was very surprised to find out they are not only allowed but encouraged, and only found that out at all when someone reverted one of my edits." "Maybe it should mention how to find sources." "I think TWA at the moment is a great steeping stone for new users such as myself. I would love to see it expand to include more 'advanced' topics that can be optionally covered by the user. I think these topics should definitely cover how to code mathematical expressions, how to find proper references externally and cite them, more detail on how to structure/format Wikipedia articles" "I would love more advanced missions. I can't help but feel that, as a beginning editor, my work is barely tolerable and likely filled with flaws or missing elements which could make it better. I know looking at other articles, especially those highlighted on the main page, gives me ideas and allows me to see examples of good work but the mission was an excellent jump start" "As I said before please cover what is notable enough to be included as an article to Wikipedia. Might be worth covering whether photos on things like facebook and instagram are considered free to be included in Wikipedia, i.e. copyright issues. Optional advance information in regards to structuring certain types of articles i.e. TV, music/dance groups, films, etc." "Formats of different articles and how to raise doubts/ask for references on other articles." "Adding images to the summary of an article and the understanding of the ideal layout for an article." "How to add photos and how to best interact with other editors when there is a dispute about content. Maybe the more inner workings of Wikipedia too - for example, how and why do some editors have more authority over content than others." "Include ways to help practice basics, because many things tend to be washed away without practice." "Eh? Maybe talk page debate basics, avoiding straw men and the like."
|
- More details: Phase 1 report
- Bot proposal
Function overview: Invite new editors to play The Wikipedia Adventure, on an ongoing basis. The results of a 4-week trial showed that The Wikipedia Adventure can be an effective tool for engaging new editors and teaching them the ropes of Wikipedia. The trial caused no disruption to Wikipedia processes and imposed no burden on editors, and it produced encouraging results. This request is to resume invitations to new editors using the same criteria as before, at a rate of ~300-500 invites per day.
Function description: Between November and December 2013, the bot was used to invite nearly 10,000 newcomers to play The Wikipedia Adventure. Several hundred newcomers followed up on the invitation and played the game. Results from a follow-up survey showed players had an overwhelming positive impression of the game, and indicate that the experience was both engaging and educational. Importantly, the invitation to play The Wikipedia Adventure caused no disruption to other processes on Wikipedia, and players were shown to be no more likely to go on to vandalize Wikipedia or be blocked from editing. Part of our success in avoiding sending invitations to bad-faith editors can be attributed to our use of data from ClueBot and Snuggle, tools which rank new editors and flag those which exhibit vandal tendencies.
We suggest that TWA could be an effective tool for supporting many new editors during their first days on Wikipedia, and addressing the ongoing decline in new editor retention. We would like to begin inviting newcomers to play the game on an ongoing basis, at a rate of approximately 300 invitations per day, using data from ClueBot/Snuggle and the same desirability thresholds as during the study. We will continue to collect data on TWA participation in order to investigate potential long-term impacts.
Function details: The sampling and data analysis plan for the TWA test will be similar to that used to evaluate the impact of participation in the Wikipedia Teahouse, which is described in the Teahouse metrics report and in this research paper.
We will invite new editors to play TWA every day. The sample will be drawn from the set of users classified as “good faith” by the Snuggle tool developed by EpochFail. A sample of Snuggle data is available here. The criteria for invitation will be:
- The user created their account within the past 24 hours
- The user has made at least 1 main namespace edit
- The user has a Snuggle desirability score of >.8. Blocked or banned accounts are excluded by this threshold, as are users who are likely to be editing in bad faith.
- The user has not yet received a Teahouse invitation.
Invites to play the game will be sent via a talkpage invitation from HostBot. Users who receive an invitation and subsequently complete at least 1 level of the Wikipedia Adventure will serve as the Experimental group (Group A). Another group of new editors who meet the criteria for invitation will not receive an invitation . Of these editors, those who subsequently make at least 1 edit to Wikipedia will serve as a basic experimental control group (Group B). We require at least 1 subsequent edit (after the hour when the user would have been invited to TWA, had they been included in Group A) in order to assure that the editors in this group would have had the opportunity to see the invitation--i.e. to make sure they had not already given up or lost interest in editing by the time of invitation. A second control group (Group C) will consist of editors who received an invitation, did not play TWA at all, but who did make at least 1 edit to Wikipedia after receiving the invitation. This control group will be used to determine whether the invitation itself has any effect on subsequent editing activities, or long-term retention, separate from the potential impact of playing TWA.The editing subsequent editing activities of the editors in Group A will be compared with those in Groups B and C. Metrics used to evaluate impact are likely to include number of edits, number of articles edited, change in Snuggle desirability score, and level of activity over time (retention), and may include other metrics.
tldr: We want to continue the HostBot 4 invites
Discussion
editSo let me get this straight -- this is just HostBot 4, but for an "unlimited" number of invites for all users that match the criteria? If so, Approved for trial (1 week). Please provide a link to the relevant contributions and/or diffs when the trial is complete. — HELLKNOWZ ▎TALK 13:13, 19 February 2014 (UTC)[reply]
- Yep, Hellknowz, exactly that, and thanks! We're just asking to extend the exact same invite process with the same criteria but on an ongoing rather than time-delimited (one month) basis. We won't be sending more daily invites than before, but we'll just be doing it continually. Thank you. Also, pinging jtmorgan about the good news. Ocaasi t | c 17:55, 20 February 2014 (UTC)[reply]
- Sweeet. First batch went out today! Ocaasi, could you watch my talk page for any questions/comments? HostBot's username is linked on the invite, and the bot's talk page redirects to mine. - J-Mo Talk to Me Email Me 01:04, 22 February 2014 (UTC)[reply]
- Alright, we've wrapped up our trial and I stopped the bot (recent invites,latest invite). Hellknowz, what sayst thou? - J-Mo Talk to Me Email Me 21:18, 28 February 2014 (UTC)[reply]
Approved. No issues. — HELLKNOWZ ▎TALK 22:24, 28 February 2014 (UTC)[reply]
- The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.