Wikipedia:The Wikipedia Adventure/Usability

Usability interviews were conducted both in person and online. The first set of interviews focused on common challenges for new editors. The second group of interviews involved the interactive text version of the script. Future usability testing will center on the online alpha version of The Wikipedia Adventure.

Needfinding

Quest(ion) Flow

Quests

Go to Wikipedia
Sign up for an account
Create your userpage
Leave me a message
Search for an article and evaluate it
Find an article’s talk page and history
Check out the NPOV policy
Add a sentence with a reference in your sandbox.
Check your watchlist
Background: how the voluntary encyclopedia stays reliable
Open questions about Wikipedia

Did you know you could edit? Why haven’t you edited before? What would it take to get you to edit?

Would you be motivated to edit a...

typo
incomplete section
wrong fact
biased article
biased article about a topic of interest to you
incorrect content about your product or company
incorrect content about you personally

What do you expect the editing experience to be like?

How do you feel about the design and usability of the site? What would you change to make it easier or better? What do you wish the site did that it doesn’t, or didn’t do that it does?

Could you have done what you did without a guide?

Knowing what you know now, how likely are you to edit in the next month?

What questions do you have about Wikipedia?

Interview one

Thoughts on vandalism in articles: "You know it's going to be fixed eventually."

If you edit: "I'm kind of old, using Wikipedia for a long time, not like I can interact with it...It's like an encyclopedia for me. I know I can edit it, but..."

What you'd fix: Wouldn't fix a joke (vandalism) but would fix an error

Sign-up: "Creating an account annoys me"

Userpage: "Creating a userpage is confusing"

First impression: "I could totally get obsessed with this. This could definitely suck me in. This could be bad."

Reliability: "If you look it up [the citation] you find out--it's real. My little sister and brother can use Wikipedia [in school]

Likelihood you'd edit in the next week knowing what you know now (1-10): 8

Could you figure it out on your own: "I'm not that tech savvy. I'm not a computer person. I'm not that confident...I feel like I'm smart enough to edit it. Unsure what would happen if I clicked edit...uncertainty."

-25 year old male from Philadelphia suburbs, week of March 25th 2013, with a Bachelor's degree and limited technical background

Interview two

How you knew you could edit: "Because it's Wikipedia"

Would you edit: "A huge amount of motivation would be needed to edit it" Not a typo, not an incomplete section, not an incorrect fact, not bias about a subject or about a topic of interest. "Only an article that impacted me personally, my company or product...something I'm personally invested in. "I'm generally a lurker."

Expectations: "I would hope that it's easy to edit."

Register an account: "I wouldn't read the sign-up instructions. I'd read as little as necessary." Username was already taken but no warning was given. "Captcha is annoying. I've seen better login flows."

Getting started tour: It doesn't mention the word edit, or 'here's how you can help'. Doesn't feel like an invitation to edit or participate.

Creating a userpage: Clicked the [start userpage link]. "Legal jargon" at the top, looks like "nothing vulgar or obscene and anything you put on Wikipedia is up for grabs for anyone". Notice the 'html' tags when bolding a word. Would prefer a WYSIWYG editor. "It feels like it could reassure me a little bit along the way. I'm not confused but I'm more engaged than I want to be. It's forcing me to pay attention and to read directions. I'm mildly disappointed in the interface. It's not the worst thing in the world. It's eh, meh. It's not like I'm going to break up with you, but... I'm not sure how long this is going to last."

Searching for a user's talk page: Search for username leads to articles not userpages. Forced to explore the navigation options. "At this point I'm lost in the abyss. This is stupid."

Leaving a message on a user talk page: "Again I'm becoming confused. I expect a "message button or a send button or a compose button." Tried the [edit] link leading to the full page markup: "I don't feel welcome here. This feels insane." Went back to talk page. "I would like to redesign this thing. 'New section' is not working for me. Put a 'send' button near the username, or at the bottom. I'm comparing this to an email model but I want this to operate like a blog comments section."

Signing a post: "In a million years would never have found the signature instructions." Then tried to sign using markup format with ~~~~ on both sides of the text. Previewed to see that it led to a double signature and removed one. "Having to sign your own posts is one more thing that you'd say, 'Why hasn't [Wikipedia] figured that out yet?'"

Article talk pages: "The talk page header makes me feel like something went wrong....I see that this is not a discussion about content but about *improving* the content. But getting that out of this was like pulling teeth". "I have no idea what 'no original research means'... oh, it's giving me instructions, guidelines."

On NPOV policy page: "I would go for the nutshell... 'objectivity' should be the goal."

How feasible would it be to do this without a guide: "Someone would have to pay me to get as far as I did. Makes me think this is some sort of non-profit thing that doesn't have resources to dive deeply into their UX."

On lack of advertisements: Prefer banner donations to "super annoying ads watching me and tracking me. Non-profit makes it a more "trustworthy 'brand'".

-20 to 30 year old male from New York, April 5 2013, with at least a Bachelor's degree and experience in user interface design

Interview three

You can edit: "I learned that is was a community based effort. It's that site every one can edit. I think is great...with many people looking at the same subject, you have a better change to get the length and breadth of the subject...the opportunity to look from various vantage points. It does credibility issues, but I notice there are checks and balances. It's a mild concern, a mild risk, not so much a concern for general information gathering--as a source I'd have more concern and want to verify it.

Go to Wikipedia: Types in Wikipedia.org. "I'm Happy that it's high up in Google search results."

Sign up for an account: "I would 'eyeball' the login info. I Glanced at the username policy, Yeah, whatever... I get that. Not going to add email now, maybe later. I am kind of protective about my email address. I'm concerned about Hacking and Phishing, through no fault of Wikipedia's, but I'm concerned Wikipedia could be a target. Captcha is not a concern, but it's an inconvenience, and we'll get past it one day with technology.

Getting started page: "It's clean and clear on 3 points with color-coded icons. If I'm editing, these are functions of contributing... Honesty, I'm a little confused. It tells me that as of having an account I have established that these are three things that I can do, but I don't know what the 3 things under the icons are [the article links]. I still don't know what the [article titles] are. I would have to click, I assume they're blue so I can click on them." Clicked through to MC Shan article. "I forgot what I was supposed to do and the articles had changed in the list Getting Started list when I went back." Used browser forward button to return to MC Shan article. "Now I see that these are articles that have the copyediting tag. It appears to me now like it's randomly giving pages that have a need for the heading [fix grammar & spelling, but the link is not clear. I wish the green pencil link was also present on the article [but the citation template does not use that icon]. The graphics and the language don't match between Getting Started and the article cleanup templates. I want to see the same icon and language. I don't want to have to keep clicking back and forth to the getting started page. I want to participate but make the edit button more inviting. I'm kind of shy, but I'm close. If you just give me a little push I might participate. The Getting Started icons I like, they're great map markers. Make the 'you can assist by editing it' be green and have the same Getting started green pencil icon, maybe at a 45 degree angle like 'here, take it, try it'. Further suggestions: Have the user sandbox upfront on the Getting Stared page, and also the watchlist. "Those are engaging. Make them very easily accessible."

Clicked edit on the MC Shan article, saw the editing page: "Oh god, you just scared me. Yikes, goodbye. Looks a little too code-y and like I might break something. Also, I'm not very connected to MC Shan. The curly brackets [header templates] are intimidating. I'm concerned about these double brackets, too. Do I need to know how to edit to contribute? I feel like I could copy and paste without breaking something, and that I might actually succeed. I feel like I could contribute to the [infobox] without breaking it. I want to make a contribution but I absolutely don't want to break something in a public space--it's like the motto, first, do no harm. It feels like there's limited places I might feel ok contributing: after = signs [in infoboxes], that seems really easy, next level is adding to a list by copying and pasting, and then, getting into the paragraphs--not so sure about that. I don't know what those double brackets are. I would have to commit my time to a learning curve that I'm not so passionate about, would determine whether or not I'd invest in learning it. I don't think they would set something up that was too easy to break, but I would expect that lacking double brackets wouldn't have the functionality. I don't know what all the apostrophe's are."

Create your userpage: "I see I don't have a userpage, but it's giving me the option to start one. It's got this sort of blank text editor. Am I creating an actual article-like page, am I going to be indexed in the search? It says userpage but it's mildly confusing. I dont know what belongs on a userpage, maybe optionally qualifying yourself (as in credentials), more information about you and what your background is. I don't know what 'watch this page' means, and I don't want to get lots of alerts. "Oh, can somebody else make a change to my userpage... maybe I have to watch this page. So now I'm making a commitment, someone could write something about me. Made word bold, "Oh good grief. I wanted it to be WYSIWYG, I wanted it to turn bold. "I'm feeling like it's not very userfriendly, it's kind of geeky. I would be willing to overlook it if I was compelled about the information... I would overcome it." Tried to undo italics, instead it inserted double italics, then quadruple italics. Manually deleted the quotations. "It seemed a little 'literal'".

Leave me a message: "I expect the talk page to have comments at the bottom, like an online article. I don't see a chat or message something. 'Talk' I would think... I don't know what that is. Talking is usually done over the phone, or in person. It doesn't feel like it has to do with written messages. I see conversation threads, I would expect to have an 'add' or a 'comment' function underneath the table of contents. I'm a little befuddled, a little lost. I'm looking for something that gives me an indication to Add something. I probably would go to the Help at this point. All it feels like I can do is edit an existing section....I don't think I want to add a new section... Edit is for something already existing." Told her New Section was the way. "I would change from 'New section' to 'New Talk'; 'section' makes it feel like I'm creating a whole new section. Creating a new talk page makes sense to me. Stick with the word 'Talk'.... I feel ok about signing my own post, but it would be a little more useful if it said 'sign your posts' near the signature button. I'm curious if it would prompt me to add a signature, but it doesn't prompt me. Hmmm... I wonder if in the preview it should indicate that the comment is unsigned. If signing is good practice, is there a reason you don't do it by default?"

Add a sentence with a reference in your sandbox: Found 'Cite', then clicked [named reference]. "I'm feeling like I need a url". Even with a url named reference doesn't work. Had to do a captcha so it saved. Told her to use the template dropdown menu. Reference worked.

Received a welcome message (unplanned): "I like it, I feel like it was standard issue, but I appreciate it." Informed it was an actual editor who did it.

What would you change: "The obvious WYSIWYG. All the fundamentals are there. No more 'confetti of apostrophes'. It feels like to be a contributor you have to be a geek. It's just polishing stuff. I want to see more consistency and leveraging common terminology. There's a very inconsistent use of Create.Read.Update. and Delete [database functions you're trying to control]]. I see different function words like 'dismiss'. That's not a huge technological thing, just consistency about the tools you already are using, renaming, repositioning--bringing them to the forefront. You have to manage real estate, find the priority functions and present different ways to execute it (links, words, icons, tabs) in multiple different ways."

--Female database designer from Philadelphia area, BS in communications, in her 40's, on April 11 2013.

Script usability testing (in Twine)

Text Usability Interview #1

Mission 1

Wasn't sure you could pick a motivation and come back to the list (will add note you can do that).

Likes the narrator's (guide's voice). Like Jimmy Neutron, boy genius

Surprised at 10 years old and 8,000 hits per second.

Most useful: tips on links and formatting.

Mission 2

Likes the Teahouse vibe. Cool that even the internet runs on Tea and Coffee.

Reply options: make a note that you can pick one but go back if it doesn't work.

"Trekker's lodge" sounds like Star Trek trekkers (too geeky?)

Mission 4

Make more clear what the skill meter is

Mission 6

Taggy was clearly "unprofessional" but at least he "admitted his mistake". "On the internet, if you call someone on their bullying they'll usually cool off, or just ignore them."

Game complete

"It was straightforward and easy to follow. I could imagine actually doing it with the editing toolbar. I liked the little jokes, they were cute. The tone makes it feel 'less like work' which is important to me. It went by a lot faster than I thought [It took 45 minutes]. Yeah, it's ok that it will take longer in the real version because becoming an editor is 'gonna take some time'. The most important thing I learned was how to edit the page."

Comments on gender gap

"I don't like putting myself out there. And I put myself down a lot. I try and keep a certain image online. Boys are more visual and get 'bold and daring' on the internet. I tend to do that more in peron. Internet actions to me lack *connection*, they're superficial."

-28 year old female with comfort using technology (not making it) and an associate business degree

Text Usability Interview #2

Likes the userpage idea, "an opportunity to express who you are"

"This is pretty cool" x3

Typofixing: "the spelling errors were funny" (laughed)

" This is great...and this is just the text part of it?? Very educational tool where you can contribute and yoru facts can be checked and rechecked. It's fun. It's almost like fact-finding mission."

"I didn't know anyone could contribute. I think it's great. The game made me feel if I had something to contribute, it would be appreciated. I think that's what most of us are after, the pursuit of knowledge."

"I'm surprised by the game being in such detail--images, comments from other editors... it looks like everyone is willing to work together, to make sure the information is correct. I think that's a good thing. It's is like a sculpture and you're able to trim away the fat to get to the heart of it."

"The game's tone was lighthearted and educational. It moves right along. I wanted to get to the next step."

--50's man, black, background in programming and technology, bachelor's degree

Alpha version

Stimulus questions for feedback brainstorm

What were your impressions, in a few words or a phrase, how would you describe the game?
What about TWA did you find 'surprising'... 'challenging'...'confusing'...'delightful'...'unclear'...'funny'...'lacking'...
What content did you learn that you didn't know before you played?
What would you change...remove...improve?
Did it leave you wanting to know more? About what?
How long did it take you to play?
Would it attract or turn off others like you?
Who do you think the target demographic was? Who do you think would most likely appeal to?
How would you describe the game's design...tone.. narrative?
Were you motivated...bored...engaged...indifferent? At which points?
Would you recommend it to a peer?
Is it a good fit for a classroom course? What level of educatino?
Would academics or professionals benefit from it? What age?
How do you like or dislike the badges, and game-like components?
Was the game accessible?
Was the game culturally, racially, and gender balanced/sensitive/neutral?

Gameplay interview #1

On the guide: “He’s pretty cool, you can tell he’s hand done…he’s [like] ‘oh hi I did you in Photoshop”
On the guide: “If you put it in photoshop and softened it a bit it would be better.”
Still fixated on the guide: “What is he…a moose? Where’s his head? This is confusing.” [much later] “OH my GOD HE TOTALLY LOOKS LIKE A PUPPET HERE’S HIS HEAD AND HIS MOUTH AND…”
Surprised at the view stats you gave (“8000 a SECOND?!?”)
“The smiley face is a bit informal…I feel like it should be more [formal] because Wikipedia is trying to be a fancy ‘pro’ encyclopedia”
It opens a new page every time you press continue, then you can’t go back and that’s annoying.
“There’s a winky face. I don’t know my feelings on the winky face.”
The userpage galactic challenge seemed super obvious/easy/geared towards 5-year-olds. (“Duh.”)
On Visual Editor “I’d always hoped they would have [WYSIWYG editing]!”
After the userpage tutorial, the guide disappeared and Marissa was super disappointed. This happened every time she saved a page.
On VE again: “This is everything I’d ever wanted when I found out [anyone could edit]…*incoherent noises*…THIS IS MY LIFE”
She recommends taking away the “what’s a good TP reply” and just have the “be friendly” notice
“Found one I like??” – the first TP message is very confusing.
The diff in Mission 4 wasn’t coded for some reason.
She thought the NPOV quiz was WAY less dumb than the other quizzes and way more useful. (Also she liked the Wikipedia idea of using acronyms for shortcuts.)
She personally really liked the “makes space unicorn cry” thing but thought it would be too weird or informal for most people.
Badges are “pretty awesome, man” – the TP quizzes were “really 5-year-old-ish” but the badges were perceived as less immature.
Quizzes in general were too easy.
The dialogue boxes like to hover over something you’re engaging with.
“Galactic Challenge!” box is funny but it pops up after you’ve already completed the challenge
The verifiability TP interaction somehow generated an edit conflict, possibly a self EC. After saving and trying to interact with the dialog box, only to be returned to the edit window she had just come from: “I’m in a loop. Sadface.”
The first TP message on M5 gave an invalid special page error.
She seemed to really enjoy it & have fun, even though it was super buggy and annoying! So that’s a yay!

--This interview was conducted by a fellow Wikipedia administrator (not me). The tester was a high-school aged female, described as "nerdy".

Gameplay interview#2

Usability Alpha Test #1

25 year old white, female college student with advanced beginner computing skills, used to teachers telling her not to use Wikipedia as a source

It's really easy to use, it directs from one step to another, makes you feel more comfortable
It's fun and simple, easy to work through; It's easing you in at a steady pace, not too taxing or stress enducing
I'm happy and excited. it breaks it down and makes it more manageable. It makes Wikipedia seem more possible.
Wikipedia is more reliable than I would have thought
The toolbar is really useful. It's not a lot of complicated codes you have to remember.
It's neat that Wikipedia has its own images
7.5 out of 10, I'm likely to make at least one edit in the next year: knowing that I can do it is definitely one of the reasons why I would now
Surprising: It's not as intimidating as I would have thought
Challenging: Nothing, easy to work through
Insightful: That you work to create unbiased articles, and the lengths they go through to check eachother--fact checking is cool
Estimated target demographic: 16 to 24, both genders, with not a lot of technical expertise
Words: helpful, informative, fun, not overwhelming, guiding, not lecturing --Ocaasi^{t | c} 22:31, 22 October 2013 (UTC)[reply]

Gameplay interview#3

Class of 5 UCSF 4th year medical students taking a course on editing Wikipedia for credit

It's well organized
It's efficient and fast
It's basic
It was good for a complete novice
One person got stuck on mission 7 adding section headings, suspected it was her error (didn't have time to troubleshoot)
It's really fun
It's one size fits all, a second version that targeted more intermediate/advanced skills would be desired
The game left me wanting to know more--wish there was more game content
It really hits values well, reinforcing the community code of conduct
Liked that there was encouragement to get answers wrong, suggested even requiring exposure to wrong answer explanations (issue: meaningful choice)
Gameplay unanimously took about 30 minutes (note, this is considerably faster than any other group, suspect these top level medical students are quite tech proicient)
Felt like it targeted a person with lower tech literacy
One person would have preferred to glean this content from a straightforward handout
One male tester felt it was targeted towards "young men", one female said there was no problem with it.
Students thought doctors would find the game useful if it was still gamified but "less childish" (noted that an academic/professional version is a potential next step)

Ocaasi^{t | c} 19:49, 19 November 2013 (UTC)[reply]