Live on Github and rule set update #1: Welcome to Ground Truth Challenge v1.1

Better Skeptics -

What a crazy 8 days since starting to hack on Better Skeptics and Ground Truth Challence v1 last Sunday! Since then we've recruited referees, got the transcripts, launched this website, got an awesome logo,  did a basic implementation of the system, appeared on Rebel Wisdom, launched the challenge, and now we're at the end of day 3!

Live on Github

As we said from the get-go, this experiment will include rapid iteration as we better understand the challenge and its interaction with the real world. As of right now, the rule set is up on Github, which will be the point of reference from now on. We've made it available under Creative Commons BY-SA 4.o as promised, so everyone can fork, modify, and run their own experiments. We will do our best to mirror the rules back to this website, but when in doubt, check Github. We hope soon to add more elements of this challenge to the repo so it approximates a turnkey challenge launch kit rather than just a textfile, as it is now.

We are also welcoming pull requests with thoughtful improvements & clarifications to the rules. We can't guarantee that we'll accept everything that's submitted, but in the spirit of the Ground Truth Challenge, what better way to challenge the process but to get down to the details and explain how you'd improve it.

Update to v1.1

Having listened and discussed with you all about what works and doesn't with the v1 ruleset, we've made a number of adjustments that should help the focus be where it should: getting to the ground truth.

You can see the exact diff with the changes on Github, but here's a high-level summary of the changes between v1.0.0 and v1.1.0:

  • Initial submission limit has been increased to 5. As in v1, each validated submission is rewarded with an extra 3 submissions available.
  • We've clarified that each submission should be stand-alone to make the job of the referees as simple and fast as possible.
  • Responses: clarified how are reviewed, how scoring is redone and how awards are determined
  • Responses will have 48h to be submitted rather than the original 24h, to make sure we have enough time to get to the best, most factual answers. The extension will apply retrospectively to submissions currently open for comment.
  • Duplicates: Fleshed out how these can be accounted for:

     - The same claim in multiple places counts as the same claim

     - Submissions targetting the same claim (whether in one or multiple locations) using different reasoning will be assessed separately. If more than one of these gets validated, the award will be given to a) the highest scoring one after the second review or b), if both have same score, to the earliest one submitted.

  • Clarified flow of the project, including role of sanity check and two phases

What's Next?

We're heartened to see various communities engage to apply more pressure to the source material but also the rebuttals being made against validated submissions.

Depending on the volume and quality of submissions coming in, we may consider extending time available to make new submissions, but this will depend on the referees and their willingness to continue to work on this, so please be nice to them and help them do their work in peace, without second-guessing and negativity, so they can continue to engage without being burnt out. It may be hard to believe, but everyone wants to get this right, and for people with that mindset to be constantly told they're doing everything wrong, it can be exhausting.

All in all though, we're thrilled with the engagement we're seeing in both phases, and hope to see even more. To see this challenge come alive is incredibly satisfying and we hope it will be the first of many to follow.