Teahouse:pilot report

» pilot planning, learnings and next steps from the Teahouse project

The Teahouse Project was piloted on English Wikipedia in the first half of 2012. The project was planned in December 2011 and January 2012, launched in February 2012, and results were measured for 13 weeks between February 27th and May 27th 2012.

This pilot project report will cover the following areas:

methodology used to plan, launch and measure the pilot
learnings from each aspect of the approach
impact towards stated project targets and goals, as well as impact on the EN:WP community
opportunities for future project development

A complete metrics report containing all data from surveys and queries used to assess impact and results of the Teahouse is reported separately.

"I have mentioned before and I will state again that I think the Teahouse is the best thing I have seen happen to the encyclopedia since I started editing." —Ryan Vesey, volunteer Teahouse host and Wikipedian with 10,400 edits

Impact towards stated goals and targets

Goal/Target	Impact
Improve editor retention among promising new editors, especially women	28% of Teahouse participants are women, up from 9% of editors on Wikipedia in general. 2 weeks later, 33% of Teahouse guests are still active on Wikipedia, as opposed to 11% from a similar control group.
Give new users a place where they can easily, safely and comfortably ask questions and receive explanations	"I liked how it was a person who answered my question just not a machine. I also liked how quick an answer was put up. I also like how they wrote my answer so I could understand what it meant." "low-key, friendly smart people; I'm learning from the Teahouse." "(Teahouse is) making something like Wikipedia so simple."
Model a social approach to new user support, help and socialization into the community distinct from existing 1 on 1 and self-support options.	"I received a number of really helpful responses from many editors about my question about translated sources." "It's good to see there are so many people willing to help each other and the Wikipedia project"
Provide a space that is specifically designed for new users which is visually appealing, interactive, and communicates a clear sense of purpose.	"I like the design and friendliness" "I like the emphasis on simplicity so new users don't get frustrated" "I love the atmosphere and how the Teahouse works."
20 hosts active throughout the pilot	24 hosts participated at any given point in the pilot, and an average of 14 hosts were active in the Q&A forum each week
75 new editors utilizing the Teahouse weekly	409 new editors participated during the entire pilot period, approximately 39 new editors participated in the Teahouse per week
50 to 100 new editors invited to participate in the Teahouse daily	7339 invites were sent total, an average of 80 per day
A measurable increase in engagement and retention of new editors should be apparent	New editors who participate in the Teahouse edit 10x the number of articles, make 7x more global edits, and 2x as much of their content survives on Wikipedia compared to the control group. 33% of these participants are still making edits 2 weeks later, and they create 2x the content of their control-group counterparts

Methodology

Setup

In a series of conversations and brainstorming sessions held at WMF with volunteers and staff about solutions to closing the gender gap on Wikipedia, Gender Gap Fellow Sarah Stierch and Head of Fellowships Siko Bouterse began to form an idea for a social cafe-like space that might help support more new editors on-wiki, in hopes that offering a different support style could encourage more women to stick with editing. Based on feedback in these sessions, we put together a project to test the idea. Research Fellow Jonathan Morgan and Designer Heather Walls were recruited to the project to lend research, analytics, usability, and design expertise, and the Teahouse project was born.

Research

Next, the research phase began. This work included:

Talking with Wikimedia Foundation teams that have worked on programs like WikiGuides and Campus Ambassadors. This allowed us to build on what had been learned already from experiences with new editor support.
Researching current and past welcome, help, and support spaces on Wikipedia. This ranged from English Wikipedia's Help Desk and Esperanza project to Portuguese Wikipedia's Café dos novatos.
Exploring academic research and other documentation about online support spaces, women's participation, and peer-to-peer versus many-to-many support systems.
Creating a series of user scenarios to describe the potential benefits to new users through interactions at the Teahouse.

All of this research was developed into an online resource on meta to provide not only clarification, but justification as to why the Teahouse would be developed into the unique space it is today.

Planning

We built the project plan on meta wiki so that we could gather feedback from interested volunteers at every point in the process. Once we'd chosen English Wikipedia as the logical place to pilot the approach, the plan was also copied to English Wikipedia for easy access by that community.

Planning started with defining clear goals, targets, and metrics to measure the pilot's success.

We then defined the kind of users we were aiming to have impact on, and the features that would best serve their needs. To help determine what were the most critical components to include in the pilot, we prioritized our set of user scenarios, and then worked back from these to a set of required features for the Teahouse.

Metrics

We built metrics into the project early and pulled them as often as possible to track activity and outcomes of the pilot and help us iterate on the concept throughout the experiment. Metrics were collected about every 2 weeks and results were published at m:Wikipedia:Teahouse/Host lounge/Metrics as well as in the host lounge.

We tracked measurements in the following key areas:

host activity (inviting guests, answering questions)
guest activity (asking and answering questions, creating introductions)
subsequent editing activity for new editors compared with control groups

Communications

Throughout the pilot, the Teahouse project team aimed to be as accessible to the community as possible. We also worked with WMF Global Communications Manager Matthew Roth to develop a communications strategy aimed at audiences outside the movement. We used a wide array of communication channels in order to reach a wide array of audiences.

Communication channels include:

Extensive dialog on project talk pages on meta, the Teahouse main page, and the Teahouse Host Lounge.
A Request for Comment was published and ran for two weeks before the pilot was launched, in order to gather input on the project. This didn't generate much discussion, however, and was soon bot-archived due to inactivity.
An easy to read FAQ was created with project overview information.
The Teahouse team was frequently found on IRC in the "official" #wikipedia-teahouse ^connect room.
A blog post announcing the Teahouse on the Wikimedia Foundation Blog. A follow up post sharing results from this report will be added in June.
Tweets by the @Wikipedia account about the Teahouse, as well as other project team member accounts. The word was spread on Twitter by users like @chrismear: "Wikipedia has a new-ish project to help first-time editors get settled in. This is very encouraging"
An on-wiki newsletter called the The Tea Leaf delivered to talk pages of over 75 Wikipedians, giving a monthly report of metrics and news related to the Teahouse.
The Signpost published several stories on the project, including an interview with the fellows and Teahouse host Writkeeper.
Sarah presented the Teahouse in a panel titled "Some of all Human Knowledge: Gender and Participation in Peer Production" at the Computer Supported Cooperative Work conference.

Roles

Members of the project team all wore many hats, but each took on primary roles most suited to our skillsets. By assembling a team where each member contributed in the areas he or she knew best, we were able to cover a lot of ground. For example, having an accomplished designer work on the design aspects of the project meant the Teahouse had great design and freed up other team members to focus on what they were best at. Some of our specific focuses were:

Sarah Stierch, Wikimedia Foundation Community Fellow. Sarah took on a Community Organizer role and focused on:

Coordinating and developing the Teahouse Hosts and Host Lounge resources
Inviting new editors to visit the Teahouse and encouraging experienced editors to participate as Hosts.
Leading by example in the Q&A forum

Jonathan Morgan, Wikimedia Foundation Research Fellow. Jonathan took on a UX Researcher/Designer role and focused on:

Compiling research findings to inform project decisions
Defining use cases, technical requirements, and user interactions through user scenarios, workflows, wireframes, and templates
Collecting and reporting metrics through surveys and database queries
Building scripts and tracking processes

Siko Bouterse, Head of Community Fellowships, Wikimedia Foundation. Siko took on a Strategy and Project Management role and focused on:

Helping the team develop an overall strategy and make data-driven decisions
Leading weekly meetings and documentation development
Coordinating between staff, volunteers and community members to ensure goals were met, resources were found, and the project remained on track.

Heather Walls, Visual Designer, Wikimedia Foundation. Heather took on a Design Lead role and focused on:

Creating the Teahouse brand and bringing Teahouse ideals to life with colors, layout, language and imagery
Designing and building Teahouse interface, pages, templates and assets
Refining and communicating about design choices to the community

Many others volunteered their time and skills to the Teahouse pilot as well:

Werdna created the Teahouse gadget that allows new editors to easily ask questions
Kaldari built some of the early tricky aspects of the templates, and developed a fallback to the Ask a Question link in the Q&A forum for people without javascript enabled.
Graham87 reviewed the Teahouse to improve screen reader accessibility
Colors were tweaked for visual contrast when User:Mdennis reviewed WCAG compliance
Teahouse hosts contributed countless hours to improving pages and processes, creating scripts, inviting guests and answering their questions

Lessons Learned

Preparing with relevant research and running metrics early and often helped us make data-driven decisions
Planning on-wiki and having good documentation helped to engage the community in the project
Regular public communications about the project and sharing updated metrics, as well as availability via multiple communication channels, helped us reach and engage with a wide audience
Ongoing coverage in the Signpost helped recruit more experienced Wikipedians to the project on an ongoing basis
Having a multidisciplinary team helped us focus on what we were each best at and accomplish a lot in a short period of time

The Teahouse Space

Design

The Teahouse design is something that makes the space unique. To create a visually welcoming and engaging space, design was incorporated into our process from the very beginning.

Concept development

The first step in design is often to create a structuring concept distilled from the intended audience, setting, and message of the project.

At the Teahouse we want to soften a user's first contact with Wikipedia, to allow new editors to connect and find community before being put-off by warning templates and technical difficulties. We set out to "reduce the learning curve for new editors," a concept from Wikimedia's strategic plan to improve interface and tools for community health.

The Teahouse is designed to give people a way to emotionally connect with Wikipedia. Decisions were made based on the following considerations:

A warm, welcoming and friendly environment
Comfortable for a wide demographic that includes as many of the contributors to Wikipedia as possible
Allow people to see themselves as part of something, as creating a community
Be simple and easy to follow
Make the process of interaction as much like (what people expect of) the internet today as possible
Both fit into the context of Wikipedia as naturally as possible, and stand out as a unique cohesive element
To embody the ethos of the Teahouse through language, and in visual and interactive aspects

To combine these priorities with the concept of a "Teahouse", we began collecting a set of colors and images. Traditional teahouse imagery was selected and a palette was constructed. The internal pattern of a traditional Japanese tea room influenced the introduction of guests and hosts to the Teahouse; guests from one side and hosts from another. The garden path is represented in the graphics derived from this image; as well as the door and roof, creating a visual parallel to a welcoming approach.

Priorities

Elements, interactions, and placement on each page were designed with the following priorities in mind:

Be clear and consistent: On every page you know you are in the Teahouse and you know its central purpose; the header of every page includes the Teahouse logo and a tagline, Peer support for new editors, part of the Wikipedia Community. From the landing page you are introduced to the three basic elements of the entire Teahouse, guests, hosts and questions. Every page has a clear set of links to each of these main pages of the project, one for guests, one for hosts, and one for the Q&A page.

Highlight community: To show visitors that the Teahouse was inhabited by a community of real, friendly people, we encouraged all guests and hosts to include a picture of themselves or something important to them in their introduction. We made sure that pictures of people appeared above the fold on key pages.

Surface activity: To generate participation and give guests a sense of what to expect, we used the front page to show guests that other people were also asking questions and introducing themselves. We displayed the most recent questions at the top of the Q&A board and made sure the space above the fold on all pages included active content, rather than just static text.

Create a welcoming environment: We used warm colors, appealing visual and thematic elements in our design and used friendly and welcoming language throughout the site to assure that guests felt comfortable and confident in participating.

Provide easy-to-use communication tools: We kept the barrier to participating on Teahouse low by trying to provide easy-to-use tools for creating a guest introduction and asking a question in the Q&A forum

Present clear calls to action: We designed each page with a clear call to action ("Ask a question", "Introduce yourself") and kept instructions as clear and concise as possible.

Early design process mockups

Teahouse front page mockup
Teahouse guest page mockup
Teahouse host page mockup
Teahouse template mockup
Teahouse Q&A page mockup

Response

User responses to our design choices have been very positive. In survey responses and other discussions, participants often mentioned the Teahouse design:

Was there anything you particularly liked about participating in Teahouse?

"I like the emphasis on simplicity so new users don't get frustrated"

"The interface is pretty."

"I like the design and friendliness"

"The interface for asking a question is very slick"

"I love the atmosphere and how the Teahouse works."

"I liked how you could look at other questions that you might have not known the answers to."

"It was warm and friendly."

"I like the name, and I really like that the language is inviting. It is unlike many of the "Help" pages that read more like a technical document with the occasional "we're here to help each other" message. Teahouse overall is very inviting and proves to be quite useful."

Was there anything that particularly surprised you about participating in Teahouse?

"it was very easy to use and beautifully designed"

"How easy it is to use"

"The interface being so simple compared to normal talk page editing"

IRC conversation with a host: [5:54pm] Writ_Keeper: Teahouse is a really cool initiative; [5:54pm] Writ_Keeper: it's surprising how relaxing the site design is; [5:55pm] Writ_Keeper: I'm not an artsy type (obviously), so I never would've thought that site design would make such a difference; [5:55pm] Writ_Keeper: but it does; [5:55pm] SarahStierch: Hell yes it does; [5:57pm] Writ_Keeper: it's funny; [5:57pm] Writ_Keeper: the biggest reason I was hesitant about the teahouse navbox script; [5:58pm] Writ_Keeper: was because I didn't want to change the design at all; [5:58pm] Writ_Keeper: so yes: well done, well done indeed; [5:59pm] Writ_Keeper: I'll go on record saying that, so feel free to quote

Build

While the Teahouse was designed to look and feel distinct from other pages on Wikipedia, it was implemented almost entirely within the standard parameters of Mediawiki as configured on English Wikipedia. The emphasis on simplicity was partially pragmatic: we were on a short development timeline and had few technical resources at our disposal. Keeping the implementation of Teahouse simple helped us to:

rapidly iterate on the space throughout the pilot period
keep the experience of editing Teahouse pages similar enough to that of editing other pages on Wikipedia, and so be a teaching tool for new editors
demonstrate the rich design possibilities afforded by MediaWiki without external addons, plugins, or engineering resources

In order to make it easier for other wikis to create Teahouses of their own, we're creating a Teahouse menu that includes the documents, scripts, interface elements and external resources we used to create WP:Teahouse.

Q&A forum

More research findings are available in the metrics report.

The only change to the core Mediawiki code, and the only non-standard functionality intended for use by Teahouse guests, was a Javascript widget on the Q&A page called the "Teahouse gadget", which was built by Werdna and supported by Kaldari. The gadget pops-up an inline edit form when the guest clicks the "Ask a question" button on the Q&A page, and posts questions as a new section on the top of the board, rather than the bottom of the page. The gadget is enabled by default for all Wikipedia users, but can be de-activated on a per-user basis on their 'preferences' page. A fallback option was provided for users who did not have Javascript enabled on their browsers, which used the default "new section" functionality.

What worked

clear call to action + easy to use gadget = easy for new editors to ask questions. Both new editors and experienced editors appreciated this aspect of the Q&A board, as several survey responses show:

Q: Was there anything in particular you liked about your Teahouse experience?

"The interface for asking a question is very slick" (Experienced editor)
"Making something like Wikipedia so simple." (New editor)

talk back templates called guests back to follow up when their question was answered, allowing the conversation to continue. 95% of new editors who responded to our survey indicated that they had received a follow-up message. This high rate of follow-up brought questioners back to the Q&A board, allowing a rich 'dialogue' to emerge between questioners and answers. Several called out this follow-up and dialogue as something that they particularly appreciated.

Q: Was there anything in particular you liked about the answer you received?

A: "Very friendly and very thorough. My question was a bit vague/large, but he/she took the time to list the steps, instances, and even the exceptions in which I wouid need to know the information he/she was providing. Great context builder."
A: "multiple, quick responses. Able to engage in dialog."
A: "Though I am, comparatively speaking, well-educated, I was was something of a slow learner to the editing process - and somewhat intimidated by it. The personal approach in an email took the sting out of it. The email communicated with me in a way that made the intricacies easier to understand, in part because I was addressed personally and felt the presence of a well-intended person behind the words."

new editors felt comfortable asking a variety of types of questions. A content analysis of 100 questions each from Teahouse/Questions and Help Desk was conducted by a WMF fellow. This analysis, which compared the types of questions asked in these two help forums over the same time period, highlights the variety of questions that guests asked on the Teahouse is comparable to other help spaces on Wikipedia.

new editors felt welcomed on the Q&A board. Again and again, new editors who answered our survey expressed satisfaction (and even gratitude) for the warm, welcoming environment of the Teahouse.

Q: Was there anything in particular you liked about the answer you received?

A: "friendly and fast"
A: "I think it's a very noble goal to get users in general to feel welcome and women in particular."
A: "Friendliness of the hosts"
A: "Editors a bit more human and polite than experience outside of teahouse"

answering questions was a draw for experienced Wikipedians. Many experienced Wikipedians, hosts and non-hosts alike, called out how rewarding the experience was of answering new editors' questions and seeing their questions answered.

Q: what was the most surprising thing about participating in Teahouse?

A: "Seeing new editors realise that we aren't all out there to get them"
A: "Being able to help new users who I then saw use the advice and do very well in their area of interest."

supportive peer environment encouraged new editor participation. new editors who answered questions expressed feeling empowered to give back to the space

Q: what made you decide to answer a question?

A: "I wanted to answer a question to help out at the teahouse, I knew how helpful Nthep and Sarah had been with my questions so I wanted to return the favor to other users"
A: "I would like to help other wikipedians with their problem and hope that we can make wikipedia a better encyclopedia"
A: "I knew them!"

guests got speedy responses to their questions and were able to find their questions again later. The median time for answering a question was 29 minutes (mean 1.5 hours). The quotes above demonstrate how much guests appreciated the rapid response, and the ease of use. We feel these findings validate our design choice of having new questions posted at the top of the board, above the fold on the page, but more user studies may be needed to confirm.

What didn't work

new users were less confident answering questions (or replying to an answer) than they were in asking them. Although many guests did post a follow-up or a reply on their own question thread, only 11% of new editors stated that they had answered a question on Teahouse (versus 65% of experienced Wikipedians). Since one of the primary goals of our project was to encourage peer support among newcomers, we want to increase this proportion significantly. We see two factors contributed to the low answer rate among newcomers:

newcomers were intimidated by the Wikipedia interface itself. 59% of new editors who responded to our survey indicated that learning the editing interface was one of the most challenging aspects of their Wikipedia experience. We believe that this underscores the importance of making it as easy for new users to reply to questions on the Q&A board as to ask one.
there weren't enough prompts in the interface that specifically encourage new editors to post answers. While there is an explicit invitation for new editors to answer questions at the top of the Q&A board, including an inline prompt (for instance, replacing the "edit" link next to the question thread with one that said "reply") could encourage more new editors to answer questions.

not surfacing host-methods to non-host Wikipedians who showed up to help out made the experience less consistent for guests. Hosts knew to add a talkback template to a guest's page after answering their question. Many hosts even had custom userscripts installed that allowed them to add in a talkback with a single click. Hosts also knew to explicitly welcome guests when answering questions. Although these were crucial features of the Q&A board that made the experience positive for guests, Wikipedians who dropped by to answer questions weren't aware of these features. One of our future design goals is to make these features more apparent to question answerers, and to include interface prompts that remind them to use a friendly tone and talkback templates when answering questions.

use of the traditional editing interface for answers resulted in edit conflicts. Because there were often several hosts watching the Q&A board at a time and waiting for new questions to come in, edit conflicts among answerers were frequent. Frustrating occurrences like this would be solved through the use of modern, forum-type software tools of which there are many open source alternatives available. These tools, however, will require integration and code review.

Intros

More research findings are available in the metrics report.

We developed templates to give new editors step-by-step instructions for creating a simple introduction about themselves on the Teahouse guest page. We also asked all active hosts to create an introduction on the host page.

What worked

users liked the social intro feature and seeing other intros. This feature confirmed the importance of seeing and being seen in a many-to-many context.

30 new editors uploaded and added images of themselves to their profiles. Others chose an avatar image and shared personal information in their intro test instead
Q: Was there anything in particular you liked about your Teahouse experience?

"Reading the other intros gave me a good feel for the sophistication level of questions that were expected. That made the process less intimidating."
"I'd enjoyed seeing others- I'd have considered those who went on before inspirational to my own effort."

a user-friendly process made creating an intro more satisfying. Step-by-step instructions and a simple workflow meant nearly 200 guests were able to create an intro.

Q: Was there anything in particular you liked about the experience of creating a guest profile?

"It was pretty easy to introduce myself."
"There was no hassle involved."
"The process is relatively simple. That's what is impressive."

What didn't work

having too many intros on a page. The number of new intros created per week dropped steeply after week 8. Once there were over 100 intros on the page, new editors may have felt less inclined to introduce themselves because theirs would be lost in the crowd.

due to limitations of template build, the task of creating an intro was still complex. Even with step-by-step instructions, the error rate of creating an intro was still fairly high, and many guests' intros needed to be manually fixed after creation by Teahouse hosts.

adding images is hard for new editors, this process still wasn't easy enough. Many new editors did not add a custom image, or upload one of themselves even though the instructions explicitly encouraged them to do so. This suggests that uploading and adding images was still something that new editors struggled with. Some new editors also tried to use copyrighted images for their 'avatars', which subsequently had to be changed. These challenges with image formatting and use are reflected in comments that new editors made in response to the survey question, what are some things that have been challenging for you on Wikipedia?

"Understanding how to post images should be reworked - too cumbersome. Apple's developer interface is much easier to use. I have ideas abut how Wikipedia could automate the process and make it elegant and simple."
"illustrating an article (don't know what happened to picture I tried to put on Wikimedia Commons, and wasn't sure whether I should try it a different way). It was just frustrating enough, partly due to concerns about copyright issues (for early 20th century art), that I haven't gotten around to it."
"IMAGE POLIFICES and LICENSE POSTINGS (Wikimedia Commons VS regular Wikipedia image upload)"

invited users were more likely to create an intro than non-invited guests 92% of intros were created by guests who had been invited; only 8% were created by 'drop in' guests of the Teahouse. By contrast, 40% of newcomers who participated on the Q&A board (by either asking a question or answering one) were drop-ins. We suspect that this disparity exists because guests who had not been specifically invited may have felt relatively more comfortable performing lower-stakes activities (asking questions) than actions that signalled group identification (creating a profile).

nothing in the interface encouraged experienced Wikipedians to welcome guests who created intros. As a result, few guests were explicitly welcomed after creating an intro, even though a welcome template existed for this purpose.

lack of suggested follow-up activity after creating profile left guests on their own. After guests have introduced themselves, there is a great opportunity to suggest activities for them to perform or people for them to interact with. Unfortunately, we did not capitalize on this opportunity. Once a guest created their intro, they ended up on the Guests page again, with no suggestions of what to do next.

Back of the House

Teahouse was divided into a customer-facing space and a "back of the house" space in order to keep things simple for new editors while giving hosts everything they needed to do their jobs well.

Database reports

We created an automated daily report of new editors who had made at least 10 edits within their first 24 hour period or who had made at least 20 edits over at least three editing sessions within their first four days. This report included links to the contribution histories and user talk pages of these new editors, as well as a link to send the editor an email if that editor had Wikipedia's "Email this user" feature enabled. Teahouse hosts used this report to select which new editors to invite. See the invitations section of this report for a more detailed description of the invite process.

HostBot

We developed a bot, HostBot, that uses scripts and database tables on the InternProxy database to perform the following routine tasks on Teahouse pages:

Generate a daily report of potential invitees on the database reports page
Regularly update the daily report to indicate which potential invitees had already been invited
Update the recent questions content block on the main page with a new set of recent questions from the Q&A page

User Scripts

Teahouse hosts also developed a variety of useful, opt-in userscripts to streamline ‘back end’ processes such as inviting and notifying guests, and facilitate easier navigation of Teahouse pages for these power-users. These scripts included

a 'host nav bar' script that added links to all 'back end' Teahouse pages (such as the templates page, the hosts lounge, and the database report page) to all Teahouse pages, including WP:Teahouse, WP:Teahouse/Questions and WP:Teahouse/Guests. This nav bar was only visible to users who opted to install the script, but made it much easier for hosts to navigate around Teahouse and access Teahouse resources, while keeping the interface simple for guests.

Host Lounge

A host lounge was built as a coordination area for hosts. See Hosts section below for more info on the lounge and host resource pages).

Wishlist

We included a 'wishlist' for project members and hosts to suggest additional functionality or features for Teahouse, or to volunteer to contribute these features. Although this list experienced relatively little traffic, we believe that it effectively communicated the project team's receptivity to community input and encouraged several technically savvy hosts to contribute to the development of useful tools and features.

Lessons Learned

Good design helped incentivize participation and define the experience. Participants enjoyed the distinctive look-and-feel, simplicity and ease of use.
Starting with clear design goals that were built off of preliminary research, user scenarios and well-defined requirements ensured we built everything with a purpose
Mediawiki is flexible and powerful for creating engaging and dynamic experiences, and we were able to accomplish more than we'd expected using the available tools. However, there are some limitations to the approach, and additional web development tools and resources would have helped make the pages even more user-friendly
The Teahouse gadget makes it easy to ask questions. Proper forum software could make it easier for both hosts and guests to answer questions.
The social intro feature added value to the space, but was underdeveloped.
We wished we'd surfaced design goals and better documented the build rationale earlier and more often. Explaining all of the thinking being each page is time consuming, but communication prevents misunderstanding.

Participation

To be effective, Q&A forums and online peer-support spaces must be populated. We knew we needed to staff the space before bringing new editors into it, and make sure there would be consistent participation from experienced editors throughout the pilot. Participation of Wikipedia volunteers and Teahouse project fellows has been critical for making the Teahouse a welcoming and helpful space. While Teahouse Hosts formed the basis for welcoming and inviting new editors to participate in the space, Wikipedia editors of all experience levels have contributed to the Teahouse's growth.

Total Teahouse participants	586
New editors who participated	398
Q&A board participants	442
Questions posted	514
Guest intros created	215
Invitations sent	7339

Hosts

Teahouse Hosts serve a critical function of making the Teahouse welcoming, active, and helpful. These volunteers are a first point of contact for new editors who visit the Teahouse. Hosts have not only provided answers and a friendly "face" to new editors, but, they have also helped improve the pilot with their opinions, input, templates and scripts.

Hosts at launch	25
New hosts joining during the pilot	17
Hosts who retired	16
Hosts at end of pilot	24
Average hosts active in Q&A forum per week	14

Initial selection

To ensure we'd have enough friendly and experienced editors staffing the space at launch, we developed an application process similar to the Online Ambassador process: potential hosts answered a series of questions about their experience in helping new editors. Stierch actively recruited editors with a history of positive Wikipedia interactions with new editors.

We had 25 applicants, and all were chosen to be the inaugural Teahouse pilot hosts.

"I really like the fact that Teahouse hosts were searched out, and the board was not just created with the attitude of 'If we build it, someone will help'." —Experienced Wikipedian, wrapup survey

Later selection

While the application process proved useful for launch purposes, it was a bit cumbersome and not transparent or scalable enough for recruiting future hosts. As the Teahouse garnered more attention on-wiki, a process for those interested in being Teahouse Hosts had to be developed. An on-wiki space (formally called "Your hosts" which is now defunct) for interested participants was created which explained the roles and responsibilities of Teahouse hosts during the pilot, listed current hosts, and asked interested parties to sign up to participate. This space and the process by which hosts are recruited and added to the project still has obvious room to grow, so that we ensure Teahouse Hosts can rely on each other to represent the project well, without creating unnecessary walls between hosts and the wider community.

Participants

Throughout the pilot a number of hosts participated in the project. The following are a list of hosts who contributed to the project (not including Teahouse project staff):

* is a Host that has retired or has taken a break from the project.

A number of Hosts came and went throughout the pilot. To ensure that the space stayed active and guests didn't find too many hosts that weren't actively available to them, we contacted Hosts after three weeks to a month of inactivity, and if they indicated being unable to continue we removed them from the hosts page. Hosts are welcome to return when they become available again.

Activities

During the pilot, the main hosts tasks were to:

Answer guest questions in a friendly, easy to understand manner that isn't loaded with wiki-speak and policy links.
Invite new editors to the Teahouse utilizing specific templates created for the Teahouse and documenting these invites in an invite tracking spreadsheet.
Monitor Teahouse discussions for any trolling or off-topic discussions, which would then be removed or moved to the appropriate on-wiki space.
Participate in the development of the pilot via talk pages on how to improve the Teahouse. Hosts contributed to the project by developing and helping with things such as:

Participation of individual hosts in individual activities and in the project as a whole fluctuated over the course of the pilot, and as some hosts dropped off others took up the role.

What worked:

populating the space with hosts selected and warmed up before launch
having clear expectations for hosts and a process to move inactive hosts out and bring new hosts in helped ensure guests always found someone to serve them.

What didn't work:

it proved unrealistic to assume that all hosts would equally handle both invitation and Q&A tasks, as these activities draw on two very different interests and skillsets. some hosts enjoyed inviting guests, some didn't. some hosts invited but didn't not answer many questions. ultimately as long as there were enough hosts to do each, that was ok, but the initial expectation mismatch did lead to some frustrations at times.

Support

Before the launch of the Teahouse, we started to build out the Host lounge, which would be further developed by the hosts themselves. This space provides tools and resources for hosts and its talk page serves as the main brainstorming and discussion hub for hosts. This space also served as a pre-launch training space.

The space is broken up into different sections:

Responsibilities which lists five key responsibilities for the Teahouse hosts during the pilot: devoting at least five hours a week to the Teahouse, inviting new editors, assisting in the Q&A board, informing the new editors that their question has been answered, and staying informed about ever changing Wikipedia policies.
Invite guide which featured the step-by-step invite process during the pilot, including how to find promising new editors (i.e. our database report), the correct templates to use to invite them, and the tracking process. Hosts also added to this page, adding their favorite venues for finding new editors.
Host tips which includes a series of tips on how to be a great Host. Tips range from remembering to say "Hi!" when new guests visit the Teahouse to avoiding wiki-jargon while answering questions. This paged served as a great tool when working with non-Hosts to understand the Teahouse methodology.
How-to guides was created by Sarah and the Hosts pre-pilot. This collaborative process allowed the Hosts to work on their skills of providing simple, easy to understand answers to common questions. Broken down into themes such as "Conflict of Interest" and "References and sources," Hosts shared their favorite responses and worked together to develop talking points for use in the Q&A form.
Templates features the many custom created Teahouse templates that Heather made in one place.
User scripts was created by Host Writ Keeper after he created a number of Host scripts to make inviting and informing new editors about the Teahouse easy.
Five pillars keeps the tradition of Wikipedia's Five pillars as part of the Teahouse, but, Sarah rewrote them to be easier to understand for new and experienced editor's alike.
Project contacts is a space that showcases the most active WikiProjects. This space is evolving to serve as one-stop shopping for Hosts and new editor's to find friendly WikiProject participants to connect with based on their area of interests. This intends on making the new editor's transition into a new project or sister project friendlier and easier.
Glossary was developed to provide easy to digest definitions of wiki-speak and jargon.
Host breakroom is the latest addition. The space is where one can find Hosts who aren't currently participating in the project.

In preparation for the launch, we started discussions with hosts about their early experiences as editors, in order to help them connect with the early experiences that inspired them to continue editing and inspire them in their host duties.

Ultimately, some areas of the space proved to be more used than others during the pilot period. The invite guide and host tips were used more frequently than the other resources. The talk page of the lounge was a very useful discussion hub.

What worked:

using the host lounge talk page to coordinate the group
host tips was a valuable tool to help set the tone of the space
invite guide was an important reference for this key host task during the pilot
how-to-guides were helpful for starting discussions with the first set of hosts, as a practice space for responses, but was not used later on

What didn't:

Most of the other resources proved to be unnecessary or weren't used, we could have saved time and not created them

Other experienced editors

In keeping with the spirit of wiki, anyone can participate in the Teahouse. Editors of all experience levels have asked and answered questions and added themselves to the guest page. When the Teahouse launched, we received more interest than expected from other experienced editors who also wanted to answer questions in the Q&A forum. This was a challenge for the project, because Teahouse Hosts had been through training about their unique roles and had agreed to do things a bit differently than on other Wikipedia pages. For example, hosts were asked to first greet and welcome guests before addressing their question and to avoid wiki-speak and linking to archives or lengthy documentation. While we assumed good faith with all contributors, many of the experienced editors, who were excited to help, were unaware of the unique style of the Teahouse.

The team wondered: How do we preserve a very newbie-friendly tone in the Teahouse without becoming a club from which some long-time Wikipedians feel excluded?

What didn't work:

trying to restrict question answering to the original hosts. reaching out directly on the experienced editors' talk pages and bluntly informing them that this wasn't how things should be done and asking them to learn more about becoming a host before participating resulted in understandable backlash and several experienced editors expressed feeling unwelcome at the Teahouse project, which went against our commitment to make it a place where all are welcome.
using a template instead of a custom message to convey the same info. The template saved some time but had little response - there wasn't backlash directed towards 1 person anymore, but most people contacted this way just ceased participating.

What worked:

adding a more friendly and personalized tone to the template. we encouraged non-hosts to learn more about the host program, and also shared the host tips page with the participants.
updating the host signup page addressing experienced Wikipedians' concerns, clarifying that non-hosts were welcome to answer questions but simply asked to learn more about the space and read the tips before doing so.
letting go of some control and leading by example. hosts consistently demonstrated the Teahouse way (we found having multiple answers per question were helpful to ensure guests did get greeted and welcomed). light but consistent reminders ("say hi!") and gratefulness for everyone's efforts proved a workable combination.

Community input

Community input continues to be key in developing the Teahouse. Throughout the crafting of the Teahouse and the launch of the pilot on Wikipedia the community participated on the meta talk page and the Teahouse talk page. Community fellows and Wikimedia staff engaged with the community to explain the reasoning behind certain aspects of the project and understand and respond to concerns about the approach on topics including:

Design and usability of the Teahouse

The Host application process and responsibilities

The invitation process

How come blocked users or bad faith editors might be invited?

Promoting the Teahouse

As this pilot period wraps up, hosts and other experienced Wikipedians will continue to be critical stakeholders in the future of the Teahouse.

Invitations

More research findings are available in the metrics report.

Personalized outreach tools are a major aspect of the Teahouse experience. These tools were used to by hosts and other Wikipedians to invite new editors to the Teahouse. Invite templates were created to have a personalization aspect and a unique color scheme that fit in with the look and feel of the Teahouse. Invitations would be sent to anyone that the inviter thought of as a promising new editor (no vandals, please!).

An invite guide was created for host use, which showcased the best ways to find and invite new users, including:

Database reports which surfaced new editors making at least 10 edits in a 24-hour period.
Feedback Dashboard which has new editors seeking assistance or giving feedback via Moodbar.
New editor contributions which showcases edits made by new editors. Hosts were asked to examine those contributions and invite promising editors.
Articles for Creation which features many good faith editors and led to the development of a custom Teahouse invite template for AfC reviewers to invite users who have had their articles declined.

These were the most valuable places to find new editors. Hosts and editors also invited new editors via their own watchlists and other means. The overall response rate to these invitations was about 4.5%.

What worked

Inviting editors when they needed help most boosted response rate. Editors invited from the Help Desk and Feedback Dashboard, and editors who had recently had their article rejected through AfC, responded at a higher-than-average rate (although only 1%-3% higher)
Hosts created scripts and custom Twinkle templates which made the invitation process easier. For instance, adding a Teahouse template to Twinkle allowed hosts to drop off invites with a few clicks, rather than manually copy/pasting a template into the invitee's userpage.

Emailing users invitations. 7.3% of emailed guests subsequently visited the Teahouse, compared with 5.1% for guests who received a template invite only. This indicates that reaching out to very new users through external channels such as email can enhance the 'stickiness' of community projects or Wikipedia editing in general.

Transcluding invite templates allowed them to be tracked automatically. Substituted templates cannot be easily located after they have been added, because they show up as regular Wikitext. Transcluding templates allows them to be tracked automatically through the "What links here?" feature of MediaWiki. Transcluding templates also allowed us to track when invites had been sent to particular users through an automated script.

Tracking invitations in spreadsheet allowed for metrics gathering about where invitees were found, who was inviting the most, and how they were invited. Tracking invitations in a spreadsheet, while time-consuming, was the only practical way to determine which sources of invitees yielded the best response rate in the early pilot stage.

What didn't work

Response rate from inviting users was low compared to investment it took to invite. Our response rate would have been higher if we had focused on guests with a higher edit count, who had been members for longer. On the other hand, that would have undermined our goal of reaching the kinds of newcomers the Teahouse was designed to support: editors who were likely to give up sooner, whether because of Wikipedia’s lack of usability and sociability or in response to early, negative experiences.

Tracking invitations in the spreadsheet was time-consuming. Considering the volume of invites necessary to get even a few dozen new editors to visit the Teahouse every week, manually tracking invites is not a sustainable long-term strategy.

A very small group of Hosts regularly invited and tracked those invites. Probably because inviting was so labor-intensive, relatively few hosts sent out more than a few invites, although more than 20 hosts sent out at least one. Despite efforts with scripts, the process was still burdening and often left to just one or two people. Several hosts have requested more automation of the invite process for Teahouse.

Student participants

Teahouse project Fellows worked with the Wikipedia Education Program to explore outreach opportunities with English language classes which participated in the Spring 2012 semester. We hoped to provide some backup to online and campus ambassadors who often were just one person supporting many students, and we also hoped to encourage and retain these often prolific and promising new editors. WMF's Global Ed team provided a list of classes that would also have a larger number of women students, allowing us to aim towards experimenting with the Teahouse as a space to support female editors. Nine classes were chosen by Global Ed staff to participate. We reached out individually to each professor to let them know about the Teahouse and offer the space to their students.

Only two classrooms appear to have sent students to the Teahouse, though, and these classes were led by professors connected with Teahouse project leaders, not affiliated with Global Ed.

What worked

Students who were actively encouraged by their professors to participate gave positive feedback on the space

It was frustrating that a comic book I was trying to cite only had its plot summary listed on a bookseller-style websites and not on, say, Google books/scholar. I went to the Teahouse to ask how and why sites get blacklisted from citation and if this was a common practice. An hour later, I got a very welcoming response from a fellow wikipedian who gave me this helpful answer..." - (female student)
"My experience in the Teahouse helped make me feel less ostracized. I was not afraid to ask a question because all of the people answering questions were being very polite and understanding." - (female student)
"I posted a question on the teahouse page... I was surprised to see how quickly I received a response! Within four hours another Wikipedian had responded to my question with a very clear and concise answer. I Found the whole process very simple, and will be sure to use this resource in the future." - (female student)

What didn't work

Emailing professors the link to the Teahouse and expecting them to pass it on to their students. Campus Ambassadors might have been a better point of contact to encourage students to visit the Teahouse
Inviting students on their talk pages didn't yield any more participants. Perhaps this is because students have different motivations for participating than other new editors on Wikipedia, or perhaps this was due to the pilot's timing in the academic calendar.

Lessons learned

Pre-populating the space with hosts who had been selected and trained ahead of time ensured a good experience for guests at launch
Striking the right balance between open-ness to anyone who wants to participate in answering questions and preserving the warm Teahouse tone of answers is critical for keeping this space alive
Personalized invitations are a useful concept, but labor intensive
Hosts have an amazing capacity for sharing their knowledge, and a speedy response time
Students who visit seem to enjoy the space as much as other new editors, but it was difficult to get them to visit with current methods
A friendly tone, prompt answers, and talkback templates bring guests back - 25% of guests are repeat "customers"

Outcomes

Engagement and Retention

More research findings are available in the metrics report.

In order to determine whether participating in Teahouse has a positive impact on an editor's likelihood of continuing to edit Wikipedia or on their other editing behaviors, we analyzed the outcomes for three groups of editors in three different 'conditions'--two different control groups and an experimental group--drawn from editors who joined between February 25th and May 19th.

We measured the level and type of activities these editors performed after the date they were invited to Teahouse. We highlight particularly notable findings from this study below. A detailed report is available in the Pilot metrics report.

Lessons Learned

Teahouse guests make more article edits, and add more article content. Teahouse guests edited 10x the number of articles as non-guests, made 7x more global edits (all namespaces), and almost 2x as much of the content they added to articles had survived after the pilot was over.

More Teahouse guests are still editing, and they have made more recent edits. 33% of Teahouse guests are still active on Wikipedia within the past two weeks (May 23rd - June 6th) vs. 9% - 11% from the control groups, and recently active guests have made 2x as many article edits on average during that period.

New Editor Experience

More research findings are available in the metrics report.

Overall, new editors really enjoyed their Teahouse experience. 71% of new editors surveyed reported being satisfied or very satisfied with the Teahouse.

New editors also had many positive things to say about Teahouse:

"I'm very happy to be a Guest here and to have the Teahouse's support in my editing efforts." —User:AsadUK200

"I like the name, and I really like that the language is inviting. It is unlike many of the "Help" pages that read more like a technical document...Teahouse overall is very inviting and proves to be quite useful." —Teahouse guest

"Overall the execution of this idea has been brilliant...the teahouse is what changed me from being a gawker into being a (very newbie) editor." —Teahouse guest

Two new editors' personal experiences with Teahouse help illustrate the potential of this approach:

Doctree's story

Wikipedian Doctree created his account in 2011. He didn't edit much at first, but in March 2012 he started to become more active. As he started to make his first edits, a welcome template appeared on his talk page. "Gad, the thing has 65 links to WP articles!" he exclaimed when reflecting on his early editing experiences in April 2012. "I felt overwhelmed rather than welcomed."

Doctree didn't let the overwhelming template deter him, instead he decided to seek help improving his editing skills so he could contribute articles as part of WikiProject Birds. He decided to explore the adoption program. Despite efforts to reach out to mentors with an interest in science, he discovered that most were inactive. While not able to find a mentor, he came across Pluma's adoptee homework assignments. He found a lot of use in this, and learned "more in one night than in two weeks of muddling through the links in the official Welcome template."

Doctree used what he'd learned to write his first article Living Bird. He then started revising the article Chimney Swift. He wrote the revision in his sandbox, and then didn't quite know how to move it out to the mainspace. Doctree found the Teahouse and asked for help. "I had one simple question: How to get the revision into mainspace? Sarah provided a simple, straightforward answer, not a referral to WP pages. Thanks. The experience you offer, as Hosts, makes a huge difference. Thanks again."[1]

But Doctree's participation in the Teahouse didn't stop with asking questions. Today, he gives back to his peers by answering other questions at the Teahouse - he's answered 13 questions so far. Doctree is a good example of the Teahouse's potential to build community for new editors and how peer-to-peer support systems like Teahouse can be self-sustaining, because as new editors get help and become more confident, they too can help others. Doctree plans to retire from his day job soon, he notes a goal on his userpage: "Retire in July 2013 and become a real Wikipedian."

Gmhayes4's story

Gmhayes4 joined Wikipedia approximately four months ago. You can call her Gmhayes4, or you can call her G, as her friends prefer. As a freelance writer and marketing consultant, G wanted write an article about a client, April Masini. Masini's article, like many articles written new editors with a conflict of interest in their chosen topic, had seen it's fair share of bad days - it was created, deleted, created again, nominated for deletion, and so forth.

Gmhayes4 appeared on the Teahouse's daily database report of promising new editors. Sarah saw how motivated she was to contribute and posted a Teahouse invitation on her talk page, as well as an email invite. Moments later, G had written back to Sarah, stating her conflict of interest and how she needed help. The article was being drafted in G's sandbox, and Sarah started working with G to learn about conflict of interest issues and teach G how to edit ethically within Wikipedia's policies of neutrality, COI and reliable sourcing.

As Sarah didn't have enough time to help G one-on-one as she would have liked, she soon directed G towards the Teahouse for further support. G was one of the first visitors at the Teahouse, asking a question about how to reference a PDF. Host Bilby jumped in to lend a hand and G was beyond impressed with the fast assistance: "Thank you so much, Bilby! That is exactly what I needed to move forward."

G continued to participate at the Teahouse. She followed other conversations at the Teahouse and chimed in when she had similar concerns or comments. With guidance and perseverance, G not only learned how to edit collaboratively on Wikipedia, but she also created a neutral, well sourced article about April Masini which has held its own (without nomination for deletion) since late March. G reflected on her experience at the Teahouse and how it helped her write a neutral article: "Hello Teahouse! Wow! I've learned so much just by reading the articles listed in this talk...[I] am having a blast creating the article! It's been a challenge to reign in my excitement about [April Masini], and stay objective in my article."

While G is busy with real life: "working, not working, reading, writing, and graphic art," she's also making edits here and there about subjects like social media. G is a great example of an editor that is often turned away by Wikipedians because of her conflict of interest with subjects. However, with patience, careful guidance, and the support of the Teahouse, which led to broader support in the community, G has become an active Wikipedian with over 220 edits in four months.

"Seeing editors who have got into problems turn themselves around and become productive is good too." —Experienced editor

Experienced Editor Experience

More research findings are available in the metrics report.

Teahouse feedback from experienced editors (Hosts and non-Hosts) has been highly favorable. 70% of experienced Wikipedians said they were "satisfied" or "very satisfied" with their Teahouse experience. Only 5% of experienced Wikipedians said they were "very dissatisfied" or "dissatisfied".

Experienced editors were impressed with the friendly, interactive and helpful landscape that the Teahouse provided:

"The friendly atmosphere - we could do with more of that on wiki."

"I liked that the Teahouse is a ground for new users so they have lots of support, and that hosts and new users interact a lot with each other."

"I believe editors are much more friendly with newcomers, its the opposite of don't bite the newcomers, its more like give the newcomers a sucker and a high five."

"I like the emphasis on simplicity so new users don't get frustrated."

"I think Teahouse is a wonderful way for new and inexperienced editors to get involved in Wikipedia while making friends with both hosts and other editors alike. Questions are answered quickly and thoroughly and most of the time the editors find the information extremely helpful."—LuK3

"The data has proven that the Teahouse is retaining editors and I think it is the friendly nature of the Teahouse that does that." —Ryan Vesey

Experienced editors also liked the design, which is a refreshing change to the standard Wikipedia fare:

"The interface is pretty."

"The interface for asking a question is very slick."

While some experienced editors liked the design, others were uncomfortable with the top-posting mechanism to the Q&A board:

"Also, as an aside, I know this is a losing battle, but the "new threads on top" thing is still a bad idea, but I know that people who are more important than me have decided that it will go that way, so I guess we are stuck with it. Still feel the need to lodge my standing complaint on that one."—Jayron32

It's also a nice change to the standard help documents:

"A very specific answer to a specific question. Sometimes the regular help and policy pages are so broad that even after hours of reading, I still didn't find an answer to a specific question."

Carolmooredc appreciated the atmosphere of the Teahouse, as well, believing it provided respite from the at times contentious culture of Wikipedia:

"A little haven of tranquility away from the battles outside. :)"

Experienced editors also found the Teahouse an important tool in improving their own skills in communication and mentorship. Gareth Griffith-Jones started using the Teahouse to ask questions and answer them, "I consider Teahouse a very valuable tool." He was happy to see his "protege" start improving their own participation skills, "my protogé/e uses the Discussion page [now], whereas the experienced user relies solely on lengthy edit summaries."[2]

"The other big plus for me has been how much I've learned either from others answers or in researching answers myself."—NtheP

Host Rosiestep noticed the successes and hopes they would be long term:

"I feel proud to be associated with the Teahouse pilot project; talk page comments from new editors abound with notes about how they appreciate the service provided. If just one of those folks becomes prolific as a result of the Teahouse encounter, then job well done here."

Gender Gap

The unique look, feel and supportive environment of the Teahouse aimed to not only retain new editors but specifically support new female editors. Women who participated in the project have responded positively to it. Although exact gender numbers are never simple to benchmark on Wikipedia, we can use survey responses as an approximation. 28% of guests surveyed indicated their gender was female. This is higher than the 9% figure currently believe to represent female editors on Wikipedia based on wider editor surveys, but its still not enough to close the gap.

Outreach

Specific tactics were used to reach out to new women editors:

Reaching out to classes with high female participation through the Global Education Program.
As some hosts scanned queries and contributions for new editors to invite, they made an extra effort to seek out and invite editors with female user names or other userpage indicators.
Social media outreach, relevant samples of coverage on Twitter.

@sarah_stierch: "Wikipedia Teahouse a warm welcome for new editors..stop by today! http://blog.wikimedia.org/2012/03/05/wik … ‪" (6 retweets)
@sikob: "Experimenting with ways to get more women involved in editing @Wikipedia. Come visit WP Teahouse! http://... ‪#Wikimedia‬ ‪#Women" (6 retweets)
@WikimediaWomen: "Just 1 way we are working 2 make @Wikipedia more welcoming for new women editors! Stop by the Teahouse, today! http://..." (1 retweet)
@refco27: "New editors shd ✔ out Wikipedia teahouse. Wikipedia's dearth of women contributors: http://en.... | ‪#techwomen‬ ‪#gendergap" (1 retweet)

Encouraging edit-a-thon participants to utilize the Teahouse during WikiWomen's History Month in March.
The presentation of the Teahouse at conferences examining the gender gap.
Mentions in blogs and in the media.
Invitation of new editors from WikiProjects such as WP:Feminism and WP:Women's history

Women's feedback

Women who viewed or participated in the project as either new or experienced editors responded positively, indicating that the Teahouse does have potential to impact the gender gap.

Editor Ria4983 shared about the Teahouse invitation she received on her university class blog. Feedback from the blog includes:

"I was contacted by some very friendly person who redirected me to something called the WP Teahouse." - Ria
"This site looks really inviting, and seems to have the potential to make learning the ropes a looked for experience." - Janice
"The “teahouse” seems to be a recognition on the part of Wikipedia that it needs to cultivate new editors in a more nurturing environment." - Meiling

Wikipedian Justinesherry wrote about the Teahouse on her blog. She shared four experiences on Wikipedia with the final being her invitation to the Teahouse.

"If you click through to the Teahouse, it’s clearly aiming to broaden female participation - just look at the pastel background and references to tea."
"However, in substance, it does a lot of good not only to make Wikipedia more inviting to women, but to new editors in general. First, it provides profiles of Wikipedia editors, aiming to present well-rounded individuals with diverse backgrounds so that any new editor can feel like he or she “fits in”. Second, it provides a Q&A for new Wikipedia users to ask questions and chat with current editors. I didn’t feel patronized flipping through it, rather, I learned some useful information - and started to feel like I would like some of these other Wikipedia editors."

Sarah presented the Teahouse in a panel titled "Some of all Human Knowledge: Gender and Participation in Peer Production" at the Computer Supported Cooperative Work conference. Sarah took this as an opportunity to share the Teahouse (pre-launch) with the audience of approximately 200 researchers. After the event, Sarah wrote that:

"As we loaded up the Mediawiki Teahouse front page jaws dropped and the over 100 attendees to the panel presentation were buzzing. A few Twitter's were sent out praising it and showing excitement. But, the best responses were in person, with the attendees who stepped up to the mic. Countless women in the audience said how much they love the idea and how beautiful it looked. At the end of the panel Jonathan and I fielded questions about the design and research that went into the concept of the Teahouse."

Twitter:

@ewingrr shared the Teahouse on Twitter: "#Wikipedia wants more women editiors. Checkout their "Teahouse" forum to support new editors http://..."
@alexishope: "cool way of retaining eds! "@chris_mcmahon: in case you've ever been intimidated editing Wikipedia, look at Teahouse: http://blog.wikimedia.org/2012/03/05/wik …"

What worked:

Response was positive when the Teahouse was shown and presented to women, and we have seen significantly more women participating in this project than on Wikipedia as a whole.

What didn't:

Because so much time and energy needed to be spend during the pilot on setting up and maintaining the space, we weren't able to focus as much as we'd have liked on gender-targeted strategies for recruiting female guests and hosts. There are clearly more experiments that need to be run in order to better integrate the space with other gender gap efforts and WikiWomen's calls to actions.
Of the outreach tools utilized, most of them yielded little return on investment in the first attempts:
- Social media did garner retweets, but, few people took it as a call to action since many Twitter users weren't necessarily Wikipedians or new editors in need of help.
- The Global Education program's female students showed little response to invites, though it is difficult to know how many of them got the information from their professors as expected.
- Despite follow-up efforts at leaving Teahouse invitations on WikiWomen meet-up participants; talk pages, few visited as a result.
- Blog and media mentions did bring attention to the project, but, it's unknown who came to the project by way of these outlets.

Female hosts

Five women participated as hosts throughout the pilot. While female hosts were highly supportive of the project, many of them did not participate as often as men who were hosts.

In discussions with hosts, the following themes emerge that may factor into female host participation in the Q&A forum:

edit conflicts were cited as possible deterrents for answering questions
with multiple hosts answering one question, there were concerns that their own answer might get lost in the shuffle and not be useful to the new editor asking the question. some women felt that this may lead to their contributions being under-appreciated,
some women were discouraged that their contributions were not acknowledged with barnstars or in other ways by fellow hosts

Women's contributions as hosts is imperative to making the Teahouse a welcoming and inclusive place for all editors, so there is clearly more work to be done in order to encourage and retain more female hosts at the Teahouse.

Lessons Learned

Women who visited the space enjoyed it, and even without much targeted outreach the participation rate of female editors at the Teahouse is higher than on Wikipedia overall. Overall, the concept of social support may be a useful tool for reducing the gender gap.
Direct support, encouragement, and acknowledgement are important tools for reducing the gender gap. Exploring more ways to acknowledge individual host contributions on a regular basis, as well as encouraging and rewarding guests for their contributions could help encourage more women to remain active in the space as well as on Wikipedia at large.

The future

The Teahouse has demonstrated potential to positively impact the new editor experience, retain and incorporate more editors into the community, and compliment other approaches to editor support in the Wikipedia ecosystem. The Teahouse project was and continues to be worth doing.

A successfully completed pilot is not the end of the Teahouse story, though. From feedback of those involved in the project, it is apparent that more work is needed in order to scale and sustain the project, solve remaining issues, add additional features, and put new systems in place to ensure that Teahouse will continue to be of value to Wikipedians for years to come. Some of this work can and should be completed by volunteers, but some of this work will benefit from continued WMF investment.

The WMF project team would like to continue to partner with the community in order to help take Teahouse to the next level. The WMF Fellowship Program intends to continue to invest in this project, support the volunteers involved, and measure outcomes over time so that we can all keep learning from the Teahouse.

Opportunites

Opportunities for continued project development under consideration include:

Automating systems that are currently manual and cumbersome >> e.g. invites, metrics, archiving guest intros
Stabilizing systems that have a single point of failure >> e.g. moving queries for invites and metrics to a stable database, building more robust volunteer processes to add and retire hosts, simplify intro-creation to not require manual design fixes, build reminders about welcoming guests into the UI, etc.
Experimenting with additional ways to scale traffic to the space >> hooks in other new-editor-facing features or interfaces, measuring carefully for capacity to serve without overwhelming hosts
Building out the social and peer-to-peer aspects of the space in ways that continue to support, not distract, from the focus on encyclopedia-building interactions >> e.g. improvements to intros, design scalable ways to reply to questions without edit conflicts, design easy ways for new editors to help each other, hooks into active WikiProjects
Building volunteer capacity and transferring admin tasks currently filled by the WMF project team to volunteers

Next Steps

During June 2012, the Teahouse team will be working on a new project plan for Phase 2 of the Teahouse project, which is expected to run July-September. This plan will be based on the lessons learned that are documented in this report, feedback from Teahouse participants, and ongoing community conversations. We hope that everyone will get involved in contributing to the post-pilot phase in order to continue to make Teahouse and Wikipedia a more welcoming and productive place for all editors to contribute to the sum of all human knowledge.

Wikipedia:Teahouse/Pilot report

Impact towards stated goals and targets

Methodology

Setup

Research

Planning

Metrics

Communications

Roles

Lessons Learned

The Teahouse Space

Design

Concept development

Priorities

Early design process mockups

Response

Build

Q&A forum

Intros

Back of the House

Database reports

HostBot

User Scripts

Host Lounge

Wishlist

Lessons Learned

Participation

Hosts

Initial selection

Later selection

Participants

Activities

Support

Other experienced editors

Community input

Invitations

Student participants

Lessons learned

Outcomes

Engagement and Retention

Lessons Learned

New Editor Experience

Doctree's story

Gmhayes4's story

Experienced Editor Experience

Gender Gap

Outreach

Women's feedback

Female hosts

Lessons Learned

The future

Opportunites

Next Steps