Imec AI research leader Steven Latré describes a benchmarking studyEvaluation is the systematic assessment of a system's behaviour or results against explicit criteria. comparing API access with running open-weight models on rented or owned infrastructureAI compute infrastructure is the hardware, facilities, networks, power, cooling, storage, and software used to train and run AI models. for long-horizon coding tasks. The study uses SWE-bench Pro and several agent harnesses.
He reports that model performance, hardware requirements, concurrency and latency vary together. In the tested configurations, some higher-capacity systems reached a lower utilization thresholdResource utilization measures how much of an available computing resource is being used over a defined period. It can describe processor activity, memory occupancy or other capacity relevant to an AI workload. for cost competitivenessAI inference economics examines the revenue, cost, utilization, pricing, and margins involved in serving trained models. than smaller systems.
Latré says agent harness choiceAn AI agent harness is the software framework that packages a model with tools, instructions, context management, execution controls, and user interaction. affected token consumption and GPU time even when task performance was similar. The examples apply to the tested coding workload and configurations, not every enterprise workload.
Watch on YouTube
