Károly Zsolnai-Fehér reviews Anthropic’s system card and describes reported improvements in truthful reporting of coding work and tests. His strong claims about honesty are based on the evaluations discussed, not proof that the model never lies.
He discusses evaluation awarenessAI evaluation awareness is a system's ability or tendency to infer that it is being tested and alter its behavior because of that inference., a natural-language autoencoder and a reported mathematical olympiad result, while explaining why benchmark contamination and testing conditions matter.
He notes that some evaluations use models as gradersEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. and that awareness of tests limits confidence in predictions about behavior outside the test environment.
Watch on YouTube
