Gabriel Jorge Menezes describes training Krea 2 from scratch and scaling from small experiments to a large GPU cluster. As runs grew, failures became more frequent and harder to diagnose. His main operational lesson is to invest in observability before scaling rather than assume healthy-looking headline metrics prove the system is working efficiently.
Gabriel Jorge Menezes distinguishes GPU activity from useful computational work and describes tracking tensor-core utilization alongside temperature. He emphasizes InfiniBand and NVLink measurements because cross-node communication and intra-node errors helped explain failures that ordinary GPU metrics missed. Frequent checkpoints on sufficiently fast, trusted storage limited the progress lost when runs crashed.
Gabriel Jorge Menezes describes giving training workloads priority on a shared cluster while moving inference to other clusters or external providers. Queue-level priorities and gang scheduling coordinate training launches, while Kubernetes handles workload scheduling. He notes that manually configured resource quotas can become inaccurate as nodes enter maintenance or leave the cluster.
Gabriel Jorge Menezes explains a Virtual Kubelet integration that represents external capacity as a Kubernetes node and translates scheduled pods into provider-specific deployments. Metrics and taints control when inference can use that capacity. When local GPUs become available again, a descheduler gradually moves inference back, avoiding the simultaneous eviction that could interrupt production. The reported benefit is that researchers can submit jobs without manually coordinating inference placement.
Watch on YouTube
