Report on the Hungarian Wikipedia experiment with partially disabling Flagged Revisions

I have translated the report on huwiki’s experiment of disabling the Flagged Revisions feature of hiding unreviewed changes. Here is the tl;dr version, see the link above for the full report:

We looked at two questions: 1) Does the partial disabling of flagged revisions help with the problem of editor decline / stagnation? 2) How much extra load does it put on patrollers? (The third question would the frequency of readers seeing vandalized content, but we did not have the capacity to calculate that.)

  • The number of anonymous edits increased by ~30%; the number of anonymous editors by ~100% (although by the nature of anonymity, the latter is less reliable). We did not see any other effect: there was no increase in the registration rates; there was a slight increase in the number of edits by active editors and by logged-in editors, but it’s hard to see it as a trend (the increase started a few months before the config change). So it seems that there is little impact on the growth of the editor community; in the future, if we find a good way to convert anonymous editors (e.g. with targeted messages after they edit), it might increase the efficiency of that.
  • The ratio of bad faith or damaging edits grew minimally (2-3 percentage points); presumably it is a positive feedback for vandals that they see their edits show up publicly. The absolute number of such edits grew significantly more than that, since the number of anonymous edits grew; the two effects together resulted in about 500 extra vandal edits and 800 good-faith damaging edits per month. The number of productive anonymous edits increased by 2000 per month. (Before disabling, the status quo way about 1000 vandal edits, 2000 damaging non-vandal edits and 7000 productive edits anonymously per month.)
6 Likes

Ciao, thanks for the update!
(For anyone who can’t remember what Flagged Revisions is, since it’s not used on all wikis, there’s a Meta page that sums it up, and the TL;DR could be that it “allows one to flag versions of articles and thus give additional information about quality. This comes with the possibility of changing what an unregistered user sees by default”.)

I haven’t followed closely in a while, so I have questions:

  1. Is your update related to https://phabricator.wikimedia.org/T209224? I see many nice graphs in there(, maybe there’s enough content for a blog post here?)

  2. What is next for the Hungarian Wikipedia with this feature?

  3. What is next for FlaggedRevs in general? IIRC, the code hasn’t been maintained for a long while…

  4. (not necessarily for you!) Are any other wikis experimenting with getting rid of it, planning to do it, or changing that mechanism with something else?

TY!

2 Likes

Yes, it is the related Phabricator ticket for this analysis. There are even more graphs here, and I gave a 10 minutes lightning talk on this topic on the CEE Meeting as well. I can imagine, that we prepare a blog post to reach more interested community members, but it is not yet prepared. :slight_smile:

We will try to reach community consensus about the future settings of the Flagged Revisions. The topic is mainly prepared for community discussion/decision, but it is not yet started.

It happens more often recently, that a new feature or bug fix breaks the functionality of the FR, so some maintenance would be useful, indeed. My personal opinion is, that FR is useful and should be kept (independently which default settings do we choose for the readers), but some technological help for the recent change patrollers would be very useful, for example integrating the ORES judgement into Flagged Revisions accepting or refusing changes where the machine learning algorithm highly confident.

3 Likes

I was active on the Russian Wikipedia when FR have been intrioduced there in I believe 2008, and I was a pretty active partoller during all the period before I left that project in 2011. My impression from that time was that the community was clearly too small to keep up with the flagged revisions, and the revisions were not patrolled for years. Sometimes young and active users decided that patrolling FR is an easy and fast way to expand theor collection of barnstars, and then they would go and do it at a rate of 10 partols per minute, which is a clear indication that the quality was not even in question. I have seen clear vandalism patrolled (actually by an administrator). So I am not really convinced, after this experience, that FR is net positive. OTOH the German Wikipedia apparently manages to handle FR backlog almost in real time, all ny edits before they made me autopatrolled were patrolled very quickly. In the Ukrainian Wikipedia, where I have made several dozen edits waiting to be patriolled, only one or two were patrolled, so that presumably they have the same backlog problem as the Russian Wikipedia.

3 Likes

Tgr’s data summarised:

Edit counts (productive/damaging/vandal):
Before, with flagged revisions: 7000/2000/1000
Without: 9000/2800/1500
Increase: 29%/40%/50%

This is interesting for IP Editing: Privacy Enhancement and Abuse Mitigation.

On Polish Wikipedia there is problem with relations between groups who think we should mark non-obvious vandalisms and who think we should mark edits thai is not only non-vandalism but quality/verified/good. We have only one flag configured and we display latest reviewed. There are voices about this discourages from editing new users. This topic regullary is discussed in Bar (also recently). We have 5-10 k of pages with unreviewed versions.

1 Like