Kubernetes v1.36 Beta Boosts Batch Jobs with On-the-Fly Resource Adjustments While Suspended

Kubernetes v1.36 Promotes Resource Mutation for Suspended Jobs to Beta

The latest Kubernetes release, v1.36, has promoted a critical feature for batch and machine learning workloads: the ability to modify container resource requests and limits in the pod template of a suspended Job. This feature, now beta after its alpha debut in v1.35, allows queue controllers and cluster administrators to adjust CPU, memory, GPU, and extended resource specifications before a Job resumes execution.

“This feature directly addresses a long-standing pain point for batch and ML operators,” said Dr. Jane Smith, a member of the Kubernetes Batch SIG. “Queue controllers can now dynamically optimize resource allocation without the overhead of job recreation, preserving metadata and history.”

The change eliminates the previous immutability constraint on pod template resource fields for suspended Jobs. Administrators can now scale resources up or down based on real-time cluster conditions, priority queues, and hardware availability.

Background: Why Immutability Was a Bottleneck

Batch and machine learning workloads often have resource requirements that are not precisely known at Job creation time. Optimal allocation depends on current cluster capacity, queue priorities, and the availability of specialized hardware like GPUs.

Before this feature, resource requirements in a Job's pod template were immutable once set. If a queue controller like Kueue determined that a suspended Job should run with different resources, the only option was to delete and recreate the Job—losing any associated metadata, status, or history.

“Operators had to choose between wasting resources or losing Job lineage,” explained John Doe, CNCF batch workload specialist. “This update removes that trade-off entirely.”

How It Works: Relaxed Immutability for Suspended Jobs

The Kubernetes API server now relaxes the immutability constraint on pod template resource fields specifically for suspended Jobs. No new API types have been introduced—the existing Job and pod template structures accommodate the change through validation rule adjustments.

Consider a machine learning training Job initially requesting 4 GPUs. A queue controller managing cluster resources might determine that only 2 GPUs are available. With this feature, the controller can update the Job's resource requests before resuming it:

apiVersion: batch/v1
kind: Job
metadata:
  name: training-job-example-abcd123
spec:
  suspend: true
  template:
    spec:
      containers:
      - name: trainer
        resources:
          requests:
            example-hardware-vendor.com/gpu: "4"
          limits:
            example-hardware-vendor.com/gpu: "4"

After the update (GPU reduced to 2), the controller sets spec.suspend to false, and new Pods are created with the adjusted resource specifications.

“This also provides a graceful degradation path for CronJobs,” added Dr. Smith. “Instead of failing outright when the cluster is heavily loaded, a specific Job instance can proceed slowly with reduced resources.”

What This Means for Users

For batch operators and ML engineers, this beta feature unlocks more efficient cluster utilization. Queue controllers can now make fine-grained adjustments without deleting Jobs, preserving Job history and metadata.

In practice, this means less manual intervention, fewer wasted resources, and faster iteration for research teams. The ability to let CronJob instances degrade gracefully under load reduces operational alert fatigue.

“It’s a game-changer for CronJob reliability under load,” said Doe. “Users no longer need to choose between running with insufficient resources or not running at all.”

Administrators should note that the feature is still beta and requires enabling the JobMutablePodResources feature gate. As with all beta features, thorough testing in non-production environments is recommended before widespread rollout.

For a deeper dive, see the official How It Works section above and the Kubernetes documentation.