Tweet

Edeb8 uses a tiered judging system. The point of this is to encourage as many as possible to have a go at judging, and provide rewards for detailed judgments. It also features a judge feedback system. This allows people to feedback to judges. As it's impossible to guess a judge's reasoning unless they provide it, judge feedback is only enabled where a judge has provided a rationale for their decision.

Of course, the system is not perfect. It depends on one very important assumption - that good judgments will outweigh poor judgments. This is actually played out in site statistics. Almost 3 times as many judgments are rated most positively as most negatively. Although this figure does not include unrated decisions, these are still outweighed by rated decisions.

It also means moderation works differently. Poor judges are given the incentive, in future judgments, to leave decisions that are worth less points, or to give better judgments. Because in principle these should be outweighed by good judgments, it's only fair that judgments are not removed from the site. This leads to additional benefits: poor judges feel more secure in their judging, and no moderator can be accused of removing judgments unfairly - the whole process is democratised.

That does not mean it is impossible for poor judges to decide a debate, however. If nobody is willing to write a high-quality judgment, then that becomes an issue. Creating incentives for good judgments is a major challenge for the site. An example of something that has been done in that regard is user badges, and more direct, public feedback through comments on judgments. It's less of a problem with the system and more with an unwillingness of people to provide good judgments.

To that end, attacking moderators because they don't remove judgments is counter-productive and goes against everything the system was designed to achieve.

Instead, the system is to create the incentives for better judgments. All it requires is for people to take part.

I'm particularly disturbed by the idea that experienced members are selectively judging debates with different point values depending on how well the side they wanted to have win performed. For example, giving a 1-point decision if the side they wanted to have win lost, and a 3-point if they could justify the win. Clearly that is an abuse of the system. Finding a direct disincentive for this, however, is more difficult. If the 1-point decisions get overruled then there is no problem, but if there is an in-group bias, then experienced members would not be providing 3-point decisions to the other side.

nzlockie missed out on a top-2 placing in the WODC last year due to a clear "vote bomb" (lowest possible ranking for a judgment) on one of his debates. I was mad then, at the site's unwillingness to judge (it was the only judgment posted on that debate). Things have improved in that department since then, at least a little.

Now the challenge is to convert 1-point decisions into 3-pointers. Rather than being met by support, recently what I've met is arrogance, claims that there's nothing wrong with these 1-point decisions, and that the system itself was flawed rather than their judging. This has in part been from last year's "member of the year".

Attacking the site moderation does not excuse one point judgments. If anything it's a thinly veiled "so what? I can do what I like if the mods won't stop me!" which frankly is a sad, sad attitude to see on the site.

I hope it goes away. And I hope that, given time, people will stop attacking the system and start thinking of constructive ways to improve it. And until that happens, do what they can to give high-quality judgments. Because the system we have right now depends on it.