Ground Truth Challenge: Results!

Here it goes!

A wee bit later than we had hoped but the Ground Truth Challenge is a wrap, all submissions have been reviewed by the referees and you can now find a detailed summary of all the challenges that made it to the second round.

In the summary, you'll find details of each submission, with a discussion of all the evidence reviewed, a conclusion, suggested rephrasing or correction of the target quote and suggested areas of further investigation.

Our verdict

So what of the outcome? In the end, only 3 challenges succeeded. Of these, two provided effective counter-arguments, not entirely refuting the target quote but providing sufficient counter-evidence to warrant narrowing or rephrasing it. The third is a straightforward factual error that was no longer true at the time of the podcast (see table below).

The material consisted of 11 hours of live, unscripted speech put under extreme scrutiny, and with no prior warning to the speakers. We therefore feel it's remarkable that so few challenges ended up as successful - however, this was our first such challenge, so we have no point of comparison.

But don't let us tell you what to think: We urge you to take a look and see for yourself, and in particular check out the discussion surrounding each claim.

In numbers

Out of 126 submissions received, 44 met our inclusion rules and were reviewed by the referees, of which 12 made it to the second round. Of these, 9 were overturned by responses, meaning they failed when referees got to examine counter-evidence, and 3 succeeded past responses.

But wait, there's more

When putting together our overview of the results, we found that for two of the claims that had failed our challenge, the evidence provided by the responses was not satisfactory (in our view). So even though those challenges failed, we feel they have merit as unsupported claims and would like to receive further information on those issues.

If you'd like to provide us evidence for or against these claims, check out the chapter "Call for further evidence" on page 3 for further info.

Note this is a 'beta' version – we're open for feedback and will make updates based on relevant or interesting suggestions. We expect a final version to be out in about a week's time, though the chance of large changes is small. We will also contact all prize winners throughout the week to coordinate disbursement of the gift cards.

Unsurprisingly, we've learnt a good deal through this process and have ideas and thoughts about what we would do differently next time. That's a topic for another time, so, stay tuned!

Resources:

• Google sheet with all submissions received, scores and referee comments
• Summary report of the results, with full discussion, conclusions and suggested rephrasing or correction of target quotes
• Our project ruleset (also on Github)
• Find us on Twitter
• Launching the $10K Ground Truth Challenge
• How does the Ground Truth Challenge work?