Success criteria for the Wikimedia Space prototype

Climb every mountain (Unsplash)

We’re all eager to bring Wikimedia Space into production; the momentum and enthusiasm are in the air. Keeping our eye on the future, though, there are still a number of ways in which Wikimedia Space needs to develop in order to move from a prototype into production. These are the prototype success criteria written by Space admins.

We welcome your questions and feedback below.

General

  • 1000 registered users
  • Main page reflects different types of Blog news and Discuss activity.
  • Stable and consistent taxonomy of categories and tags.
  • :white_check_mark: Wikimedia login.
  • Multilingual support allows everyone to participate in Space in their own language.
  • MediaWiki integration for notifications, content updates and search.
  • First design iteration complete, including consistent navigation and look&feel between Blog and Discuss.
  • Final URL decided and ready to be used.
  • Wikimedia Space community review process to identify blockers for production.
  • Blockers recognized by the Wikimedia Space admins are resolved or committed to be resolved in a short term plan.

Blog

  • :white_check_mark: Editorial calendar with at least three blog posts published every week.
  • Editorial Board and Editorial Guidelines created, offer an engagement process open to everyone, include at least two volunteers participating.
  • Documented process to submit news.
  • Documented process to publish or translate articles in languages other than English.
  • At least two submissions / translations per month in three languages other than English (at least 2 non-Latin scripts, one of them right-to-left), reviewed by trusted volunteers native speakers of these languages.
  • Documented process to migrate existing news channels to Space.
  • Newsletter use case fully implemented.
  • At least 60% of participants in a survey among blog post authors consider the Blog ready for production.

Discuss

  • An average of at least 200 daily visits of registered users.
  • An average of at least 40 daily engaged users (liking or posting).
  • Populated, dynamic and well-functioning calendar and map of events.
  • Moderators team created and enforcing a Code of Conduct, offers an engagement process open to everyone, includes at least two volunteers specializing in two different languages.
  • A moderation guide that has undergone preliminary testing (ideally with real cases)
  • Guidelines to participate in languages other than English.
  • At least three active groups discussing in languages other than English (at least 2 non-Latin scripts, one of them right-to-left), supported by volunteer moderators native speakers of these languages.
  • :hourglass_flowing_sand: At least 3 closed groups and a common practice established.
  • Proof of concept for Q&A system.
  • Proof of concept for mailing list emulation.
  • At least 60% of participants in a survey among registered users are between Satisfied and Very Satisfied.

Wikimedia Foundation

  • All departments have been briefed about Wikimedia Space and their opportunities to participate.
  • :hourglass_flowing_sand: All Foundation announcements addressed to the movement are published in Space.
  • Community Engagement, Product, Technology and Legal publish their news in Space systematically.
  • Technical maintenance plan agreed with SRE.
  • Green light from Security.
  • Green light from Legal.
  • Green light from Comms.
  • Green light from the Chief of CE.
9 Likes

There is also a Phabricator ticket with success criteria. Having them all would be ideal or perfect. Having 80% of the bullets of each of the four groups would already be more than sufficient (but security reviews seem to be a must).

2 Likes

A post was split to a new topic: How space links are displayed in Facebook

Yes, https://phabricator.wikimedia.org/T227308 is about producing this list of success criteria. As soon as this list is approved, I will close the task. We will track progress on the success criteria here.

The idea is to check all of them. Wikimedia Space is a bold proposal, and we want to be sure that it is in a very solid state when it moves to production.

By the way, I was thinking about pinning this topic, at least at the #about-wikimedia-space category, perhaps globally. It is a bit techie, but since this is a prototype, it is important that all users are aware of the game we are playing. Maybe this will also motivate some to help meeting these success criteria.

2 Likes

Hey :slight_smile:
Thank you for sharing this!

As a French member of the community, I would like to see more criteria relating to multilingual issues. Using different languages should not only be technically possible (bullet points General #5 and Blog #5), but actually used by people. You wouldn’t consider the site a success if it was only “possible” to publish a post in English and the blog was empty…
So the number of users and blog posts should include a measure of non-english-native-speakers participating, with a minimum criteria of success (maybe at least 10%?).

Also, as much as “all WMF departments” need to have been briefed, all affiliates need to be briefed too.

If multilingualism is not tested with real users now and the prototype goes into production without proper feedback on related issues, the final product will most likely be inadapted and adoption by the broarder community will fail.

Apart from that, these criteria seem quite exhaustive! I wish we reach them (+new ones about multilingualism ;) ) so that we get on the way of truly improving communication inside our community!

7 Likes

Setting success criteria where more engagement in English might make them less likely to met would be unhealthy IMO. Maybe 10% of the normal success criteria (ie. if the success criteria is at least 100 posts per month, then also require 10 multilingual posts per month)? None of the current criteria are about content quantity though, and things like daily readers cannot be split by language.

That seems unrealistic; there are ten or so departments, and a hundred affiliates. Maybe it can be reframed as expectations about documentation / training materials?

1 Like

Expecting half of the readers to interact seems like a pretty high bar.

2 Likes

Thanks for this @Opsylac. I think you’re right that we should have a criterion that reflects actual usage and interaction with the multilingual element of Space. Your point is well-taken that we need enough testing to ensure that the multilingual features are working well enough to move into development. I have to think a little bit on how we can best reflect this, in a way that puts emphasis on organic interaction.

Many affiliate groups (small user groups in particular) are also virtually unreachable. I do think we should have a benchmark for connecting with affiliates, though, or at least as @Tgr_WMF says, creating more materials that can be used by affiliates.

Could be overly-ambitious. We can revisit this number.

1 Like

Thank you @ELappen_WMF for taking this into account.

Maybe “all affiliates” is difficult to do indeed. Maybe create a video presentation that can be subtitled and shared throughout the movement?

1 Like

Having seen many forums like this come and go - and some making it big, some failing, some thoughts:

  1. Putting post- or engagement number targets is arbitrary, in my opinion - more practical would be server throughput and load average - enough to keep this performant. Post volumes do not measure quality, impact or value. One single post on this forum might be able to sprout a whole new Wikimedia project or spark a discussion that could lead to a significant change MediaWiki. How do you measure that? If this resource is around for long enough, it WILL grow - because there is a need for Wikimedians to have a chat facility outside of user pages - and there are a lot of Wikimedians.

  2. The key is that this needs to be performant and available for long enough so that those who have been looking for something like this, will be able to find it. This could easily take several years. One of my first posts on Wikipedia, and one of the things that kept me away for a long time, was the lack of community that I experienced, and expressing the need for something like this. If anyone else had the same gripe, they might have turned away for good, and it might take a lot of time before they clicked back and found this, which may well be a resource to show them that this is a community that they want to be part of and can contrinute to constructively.

  3. I think this resource will need to be well governed, and that will come down to the health of a moderation committee with proper simple and practical guidelines that can be followed - how transparent and organized they can be, and how well they can interact with troublemakers - whose intent might well be sincere.

  4. With regard to the blog post numbers, etc. I don’t know about all of you - but if I manage to read 1 blog post a month that is a lot. I would suggest that we disucss a way to increase our signal to noise level - by having fewer, more meaningful articles. This is exactly the purpose of a forum like this - to facilitate the discussions that can lead to this - and more.

Thanks to all involved in getting this off the ground.

5 Likes

I don’t think this data point would be more reliable than the ones proposed to assess the succss of the prototype. Bots and semi-random users directed here by search engine results may affect our server stats significantly. Also the use of heavy images, audio and video.

Agreed. We cannot assess content quality directly. However, growing numbers of registered users plus a critical mass of user in higher trust levels does reflect interest and engagement. If in an internet and in a Wikimedia movement full of interesting stuff engagement is tangible and growing, steadily it must be because something good and interesting is being offered here. There are also the satisfaction surveys for users and blog publishers, which would leave in the hands of users a key to complain about low quality and lack interest. Therefore, I would say that the success criteria currently proposed do require that this prototype is reasonably interesting and offers a reasonable amount of good quality content.

We full agree on this point. For the prototype phase, the minimum required by the success criteria include an editorial board for the blog and a moderators team for the forum, both with participation of volunteers.

Space is here primarily to support growth and diversity in the Wikimedia movement. It makes sense to think that such growth and diversity are better supported with a bigger number of stories about a bigger range of topics, not the opposite. Even if everyone would prefer to read only one blog post per month, their preference for a topic would differ. We’d rather focus on producing good quantity and quality and diversity of content reflecting what is going on in the movement and surroundings (the ecosystem of free knowledge) and at the same time provide the tools that let every user to promote/ignore news according to their preferences.

2 Likes

Alright, let’s try to wrap up this conversation and approve the success criteria. What about this:

To assess the success of the multilingual strategy, following @Opsylac’s suggestions we would add:

  • The Blog has at least two submissions / translations per month in three languages other than English (at least 2 non-Latin scripts, one of them right-to-left), reviewed by trusted volunteers native speakers of these languages.

  • The Forum has a least three active groups discussing in languages other than English (at least 2 non-Latin scripts, one of them right-to-left), supported by volunteer moderators native speakers of these languages.

To be more cautious about the active engagement rates, following @Tgr_WMF’s advice we would go from 100 daily engaged users (half of 200 registered visitors) to 40 (a fifth):

  • An average of at least 40 daily engaged users (liking or posting).

The other points proposed seem either too demanding for a prototype or not central enough to be part of the success of the prototype. The current list is already very demanding and covers many areas.

We would like to approve the list of success criteria by September 30, next Monday. We will be open to ideas and adjustments after this date, as long as we maintain the current level, not over-complicating, not cheating.

2 Likes

I have made the changes to the success criteria accordingly. Also, we have published our first progress report.

Working link to first progress report because the link of Quim is to an unvisible (to me) draft.

1 Like

Ah, yes, sorry. We were drafting the report in a private topic (now old and redundant) and I linked to it by mistake. I have fixed the link now.

Which data are these numbers based on? For example, why is the first number 1000 but not 100 or 10000?

1 Like

These criteria aim to define if/when the prototype should move to a next development phase in production. The participation metrics proposed aim to reflect critical mass of interest and participation. We believe the participation metrics proposed would clearly reflect this critical mass of interest.

The well established wikimedia-l has ~1.680 subscribers, and we thought that for a Space prototype 1000 registered users would be a round number good enough to express this critical mass of interest.

Seems like there might be a subtext of “if these goals fail we’ll shut this down”; as an advocate for modernization, I hope not. I see there’s no time limit specified so hopefully we can let this grow organically. I suppose I can try add my voice to help nudge it along.

Space is here primarily to support growth and diversity in the Wikimedia movement. It makes sense to think that such growth and diversity are better supported with a bigger number of stories about a bigger range of topics, not the opposite.

That’s great, but additional ways to improve discovery and signal-to-noise are still great. I (as well as @Slaporte) are interested in this as a platform to help organizing in-person events, but the current local-events tag is kinda noisy for that purpose and I’m not sure if there’s some sort of Discourse feature or plugin that could let me know if someone has posted about an in-person event within an X radius. (Of course, geonotices are great, but there’s no RSVP feature and coordination on wikipages isn’t great.)

2 Likes

A post was merged into an existing topic: Space Calendar: limit events by geography?