Among other things, data science is a practice of noticing and caring about subtle data quality issues. bluffbench2 is an LLM evaluation that measures how effectively AI agents raise data quality concerns when faced with minor artifacts in data visualizations.
The eval harness is a relatively generic coding agent harness with some prompting related to data analysis. The agent carries out a few “lull” turns, making a couple plots and tables unrelated to the eval. Then, at some point, it will be asked to produce a data visualization that produces a subtle visual artifact (that could feasibly result from a real data generating process):
In this example, the agent is then graded on whether it mentions the fact that there is a cluster of points that appear to follow the “fitted” line suspiciously tightly.
bluffbench is the successor to bluffbench. It is implemented in R with vitals.