Jason Goodison describes General Compute’s approach to AI inferenceInference is the use of a trained model to produce an output from supplied input., separating prompt prefill from token decoding and using different hardware for each workloadAI compute infrastructure is the hardware, facilities, networks, power, cooling, storage, and software used to train and run AI models.. He presents the company’s approach as an alternative for latency-sensitive use cases.
Jason Goodison discusses fast inference for coding and voice agents, including the tradeoffs between response speed, model capability and data handling. The conversation also considers capacity constraintsAI compute capacity is the available ability of hardware and supporting systems to perform AI training or inference work over time. and potential pricing for faster tokens.
Jason Goodison predicts that demand for low-latency inferenceLow-latency AI inference produces a model result quickly enough for an interactive or time-sensitive application. will increase as hardware becomes more available. The interview includes company announcements and forecasts, which are presented as his perspective rather than independently measured results.
Watch on YouTube
