Knowledge Engine (search engine)

Knowledge Engine (KE) was a search engine project initiated in 2015 by the Wikimedia Foundation (WMF) to locate and display verifiable and trustworthy information from public-information sources in a way that was less reliant on traditional search engines. It aimed to allow readers to stay on Wikipedia.org and other Wikipedia-related projects when looking for additional information rather than turning to proprietary search engines. Its goal was to protect user privacy, to be open and transparent about how its information originates, and to allow access to related metadata.

The search example of the Knowledge Engine states "Ad-free, secure, non-profit: Make Wikipedia your default search".

The project was controversial internally, and ended early in 2016. Related ideas were applied to the internal cross-wiki search engine for Wikimedia projects.

History

In 2015, WMF applied for a $250,000 grant from the Knight Foundation to support development of the Knowledge Engine. Its grant proposal noted: "Commercial search engines dominate search-engine use of the Internet, and they're employing proprietary technologies to consolidate channels of access to the Internet's knowledge and information." The project was designed in four stages, each scheduled to take about 18 months.

The project planned to draw information from Wikipedia-related projects and eventually to search other sources of public information such as the U.S. Census Bureau. Leaked internal WMF documents stated the "Knowledge Engine By Wikipedia will democratize the discovery of media, news and information—it will make the Internet's most relevant information more accessible and openly curated, and it will create an open data engine that's completely free of commercial interests. Our new site will be the Internet's first transparent search engine, and the first one that carries the reputation of Wikipedia and the Wikimedia Foundation." The new search engine was not expected to immediately replace a general purpose search engine because at first it would only draw on information from Wikipedia and its other free knowledge projects, though it might in time also have included academic and open access sources in its search results. Matt Southern in Search Engine Journal attributed media confusion about the Knowledge Engine's scope to the fact that later WMF statements clarifying the organization's intentions were "quite a contrast to the original grant application documents".

The project was not discussed publicly with the Wikipedia community while developing the concept, nor part of the existing annual plan. This secrecy was mirrored by a degree of confusion within the organization, and seen as at odds with the goal of transparency. An initial blogpost by WMF Executive Director Lila Tretikov about the project did not address why the original proposal was so much broader than an internal search engine. Some staff and WMF board members felt the WMF was still not being straightforward with the Wikipedia community. This led to a crisis for the organization, leading to Tretikov's resignation in February 2016.

Design

Example of federated data sources potentially used by the Knowledge Engine.

The Knowledge Engine had been intended to supplant proprietary search engines, instead showing how a piece of information originated and allowing access to metadata. It would not have had advertisements, and it would have protected users' private data and emphasized collaboration. It would have drawn information from Wikipedia-related projects, as well as potentially other sources of public information such as the U.S. Census Bureau, OpenStreetMap, the Digital Public Library of America, and external sources like Fox News. Jimmy Wales and the WMF stated that the project would focus on improving search on Wikipedia and related Wikimedia projects. The grant application stated that it would "create a model for surfacing high quality, public information on the internet", competing with commercial search engines.

Development timeline

Information about the project was publicized gradually. As early as May 2015, community members asked about the concentration of staff in a new "Search and Discovery" department, though public plans made little or no reference to this work. The grant was applied for in mid-2015 and awarded in September, but only publicly announced in a January 2016 press release.

The project plan had four stages, each scheduled to take about 18 months: Discovery, Advisory, Community and Extension. The initial stage of the project was budgeted to cost $2.5 million, with the whole running to the tens of millions. After a year, the WMF was to evaluate development to date, and at the close of the grant, set plans for the project to continue to the second stage.

Motivation and scope

Since mid-2012, Google Search has included fast facts from Wikipedia articles on its search results pages via the Google Knowledge Graph.

A central source of confusion for the project was the extent to which it would directly compete with traditional search engines as a place to search the Web. According to Vice, "the Wikimedia Foundation, the nonprofit that finances and founded Wikipedia, is interested in creating a search engine that appears squarely aimed at competing with Google." According to The Guardian, "there was considerable doubt over what the tool was actually intended to be: a search engine aimed at halting a decline in Wikipedia traffic sent by Google, or simply a service for searching within Wikipedia?"

Since 2012, Google Search and other search engines had started highlighting brief informational summaries from Wikipedia in knowledge panels alongside search results, reducing traffic to Wikipedia from those search engines. According to Search Engine Watch, this led to a battle for attention, and this project could have recouped some of that traffic.

Leaked internal documents from early concepts framed the plan more boldly than the final public description. They said the "Knowledge Engine By Wikipedia will democratize the discovery of media, news and information—it will make the Internet's most relevant information more accessible and openly curated, and it will create an open data engine that's completely free of commercial interests. Our new site will be the Internet's first transparent search engine, and the first one that carries the reputation of Wikipedia and the Wikimedia Foundation."

The apparent contradiction between different descriptions of the purpose led to confusion in the media and in the community. In response to speculation, the WMF published a response clarifying its intentions: "We're not building a global crawler search engine ... Despite headlines, we are not trying to compete with other platforms, including Google. As a non-profit we are noncommercial and support open knowledge. Our focus is on the knowledge contributed on the Wikimedia projects. ... We intend to research how Wikimedia users seek, find, and engage with content. This essential information will allow us to make critical improvements to discovery on the Wikimedia projects." Director of Discovery Tomasz Finc added "we are building an internal search engine, and we are not building a broad one." Jimmy Wales stated that suggestions that the WMF is creating a rival to Google are "trolling", "completely and utterly false", and "a total lie", while allowing that the Knowledge Engine might in time include academic and open access sources in its search results.

Matt Southern in Search Engine Journal attributed media confusion about the KE's scope to the fact that this was "quite a contrast to the original grant application documents", an assessment echoed by James Vincent in The Verge, Matt McGee in Search Engine Land, and Jason Koebler in Vice.

Controversy

Many in the community were furious that details of such a large project had been withheld by an organization that prides itself on radical transparency. Wikimedia's public story—that it was never working on a search engine—was directly contradicted by a grant proposal made to the Knight Foundation and leaked internal documents.

 —Jason Koebler, Vice

Large-scale WMF projects are almost always discussed publicly with the Wikipedia community, but this did not happen with the Knowledge Engine development. Wikipedians were unaware of the existence of the project as a concept, and the KE project was not mentioned in the WMF's annual plan. According to the English Wikipedia's community newsletter, The Signpost, some community members expressed outrage at the perceived secrecy around it and their lack of ability to give input, and this raised questions about WMF's commitment to transparency with the Wikipedia community.

James Heilman, a member of the WMF's Board of Trustees, noted in The Signpost that while on the Board, he had insisted multiple times that the grant documentation be made public, without success. He was dismissed from the Board in December 2015, and it was suggested that his push for transparency concerning the grant had been a factor in his dismissal—a suggestion rejected by Jimmy Wales. The Wikipedia community re-elected Heilman to the Board in 2017.

Ruth McCambridge said in Nonprofit Quarterly, "Wikipedia editors have been requesting from December for the grant proposal and grant letter for a project that many surmise is a bid to remain technologically cutting-edge by the Wikimedia Foundation, but which may divert resources and attention from other pressing needs of the community."

Commenting on the reluctance to share the grant documents with the community, referencing privacy concerns, McCambridge saw "a major difference in culture and values assumptions" compared to previous Wikimedia practice. McCambridge said that "the power of important strategic decisions" here seemed to rest "between funders and the top of the organizational hierarchy" and was "not shared with volunteer editors."

The WMF initially published only portions of the grant documentation, later making the full grant agreement available in February. Further internal documents were leaked shortly after. The full agreement clarified the initial concept for the first stage of the project. Tretikov said she regretted being so late in informing the Wikipedia editing community about the grant.

Longtime Wikipedia editor and journalist William Beutler told Vice Magazine's Jason Koebler, "Leaving aside whether a search engine is a good idea, let alone feasible, the core issue here is about transparency. The irony is that the Wikimedia Foundation failed to observe one of the movement's own core values ...." UK Wikipedia editor Ashley van Haeften told Ars Technica via e-mail that "Lila, Jimmy, and the rest chose to keep the project and the Knight Foundation application and grant a secret until the projects were underway for six months, and even then this only came to light because it was leaked."

Tretikov's initial public post about the Knowledge Engine project did not explain why the original grant proposal had such a grander vision than the later public plan to develop an internal search engine. Staff who had been uncomfortable about the project's development felt the WMF was not being sufficiently straightforward with the community. According to statements posted of an internal meeting on the WMF's website, a member of the Discovery team member said to Tretikov, "My concern is that we still aren't communicating it clearly enough. This morning's blog post is the truth, but not all of the truth. Namely that we had big plans in the past. It would have been much easier to say that we did have big plans, but they were ditched ... we still haven't acknowledged it. We can't deny it."

Former deputy director of the WMF Erik Möller, up to April 2015, portrayed the events as "very much out of control" and "a crisis." Disagreements about the project, and the response to the resulting controversy, led to many WMF staff members departing, culminating in Tretikov resigning on February 25, 2016.

External links

• Discovery homepage on MediaWiki.org