The AI Automators compare 13 decision models across 7,671 cases on a local workstation. The tests cover classificationClassification assigns an input to one or more defined categories using rules or a learned model., multilingual choices, document relevance and conversation decisions without task-specific fine-tuning. Reported overall accuracy puts hosted Jev first, followed closely by Winnow and Decider, while individual task results vary.
Unlike an autoregressive chat model, a decision model scores a supplied set of answer optionsA decision model selects or scores possible answers for a defined decision.. A small Laya model leads the news-classification subset, while multilingual performance and policy-heavy support questions favor different models. Option-count and context-window limits prevent some models from handling every test directly.
Class imbalance makes accuracy alone misleadingEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria.. Always rejecting documents scores highly on the relevance set while missing every relevant document. The AI Automators compare recovered relevant documents and false positivesA false positive is a result that incorrectly identifies a condition or category as present, such as flagging an irrelevant document as relevant. instead, illustrating why thresholds and the cost of different mistakes matter.
Short requests are faster on some local models, but that advantage reverses when the full policy handbook is supplied. Long-input processing and GPU memory requirements change the comparison substantially. Retrieval of only relevant policy sections is suggested as one way to reduce input cost.
The AI Automators recommend choosing by privacy needs, request length, hardware and measured error costs rather than a single leaderboard. The results are a creator-run comparison on particular datasets and hardware, not proof that one model will dominate every deployment.
Watch on YouTube




