Field note · JZ-006 · Behavior ·
Can your AI survive one mistake?
- The guess
Better tools get forgiven: a model that is right more often keeps its users.
- What the research found
In forecasting tests with a bonus for accuracy, people who watched a statistical model make mistakes chose their own judgment instead, though the model was better: their own forecasts had 15–29% more error. A follow-up study found the way back: when people could adjust the model's forecast, even slightly, they were far more willing to use it.
- Why
The same mistake cost a model more trust than it cost a person. A small say in the result was enough to bring people back.
- What to do
- Let people adjust what the system gives them: edit a suggestion, nudge a forecast, re-rank results. Even a small degree of control helps.
- Design the moment the system is wrong before it happens: acknowledge it, make correction easy, keep the person in charge.
- Set honest expectations about accuracy up front.
- What not to do
- Make the output take-it-or-leave-it.
- Promise a model that never errs. One visible mistake costs more than a modest claim.
- Limits
- Lab forecasting tasks with a bonus at stake, not everyday products.
- The studies measured whether people chose the model, not how they used it over months.
- Sources
Dietvorst, B. J., Simmons, J. P. & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126. doi.org/10.1037/xge0000033
Dietvorst, B. J., Simmons, J. P. & Massey, C. (2018). Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science, 64(3), 1155–1170. doi.org/10.1287/mnsc.2016.2643
Also as markdown for your AI: softwarebehavior.com/notes/jz-006.md
Use with your AI · skill
This note is also a skill.
Install it once. Your AI applies this research when it designs, builds or reviews, and cites the source.
studio-jz-algorithm-missv1.0.0
Your AI uses it when it designs recommendations, forecasts, rankings, AI suggestions and generated drafts, autocomplete, and how a product behaves after its model gets something wrong.
CC BY 4.0 · Checked against the full paper · Works with Claude, ChatGPT, Copilot, Gemini and Cursor
Newsletter · monthly
One Field Note a month.
What changed in the book, the latest episodes, one framework in practice.
Opens a message to joost@studio-jz.nl in your email app. Send it to subscribe.