A helpful reader informed me that the comment spam protection plugin was breaking after the first failed attempt.
IE if you typed in the wrong answer, hit ‘post comment’, went back and then typed the correct answer, you still got the same error message saying it was wrong.
I wondered if this was an intentional behaviour to stop automated brute-force attempts, although that would be somewhat inhibiting to people who simply made a mistake.
Anyway, turns out that it was not that, it was something in W3 Total Cache that was affecting it. Not sure what, but disabling the plugin solved the issue.