Technical Insight Explanation

Building Custom Kubernetes Scheduling: Extenders vs. Plugins

Understand when Kubernetes scheduler extenders make sense, how they affect the scheduling path, and when the Scheduling Framework is the better choice.

Building Custom Kubernetes Scheduling: Extenders vs. Plugins
On this page

Kubernetes schedules Pods through feasibility checks, scoring, and binding. Native node affinity, topology spread constraints, taints, tolerations, and resource requests cover most placement needs.

Some decisions depend on external information: accelerator licenses, for example, or rapidly changing infrastructure policies.

Two approaches are scheduler extenders (external HTTP services) and Scheduling Framework plugins (in-process extensions).

Start with the scheduling primitives you already have

First exhaust declarative options: node affinity for GPU models, device plugins and extended resources for hardware capacity, and topology spread constraints for failure domains.

If a policy depends on external state, consider synchronizing that state into Kubernetes before adding synchronous scheduling calls.

Where an extender fits

A configured extender is an HTTP(S) service that can filter or score candidate nodes and optionally participate in preemption or binding.

Diagram source
flowchart TD
    A[Pending Pod] --> B[Scheduler filters candidate nodes]
    B --> C[Extender filter endpoint]
    C --> D{Feasible nodes remain?}
    D -->|Yes| E[Scheduler scores nodes]
    E --> F[Extender prioritize endpoint]
    F --> G[Select node and bind Pod]
    D -->|No| H[Unschedulable handling and possible preemption]

This simplifies the scheduler’s actual cycles. An extender is not an admission webhook; preemption calls occur only under specific conditions.

Extenders avoid custom scheduler binaries but add a synchronous network dependency.

A concrete use case: accelerator entitlement

Consider GPU nodes whose use depends on accelerator-pool entitlement. The scheduler sees GPU capacity but not the external entitlement decision.

For illustration, compare the Pod annotation scheduling.apefactory.com/pool with a trusted node label of the same name. This does not verify external entitlements.

package main

import (
	"encoding/json"
	"log"
	"net/http"
	"time"

	corev1 "k8s.io/api/core/v1"
	extenderv1 "k8s.io/kube-scheduler/extender/v1"
)

const poolKey = "scheduling.apefactory.com/pool"

func filter(w http.ResponseWriter, r *http.Request) {
	if r.Method != http.MethodPost {
		http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
		return
	}

	var args extenderv1.ExtenderArgs
	if err := json.NewDecoder(r.Body).Decode(&args); err != nil {
		http.Error(w, "invalid request", http.StatusBadRequest)
		return
	}
	if args.Pod == nil || args.Nodes == nil {
		http.Error(w, "pod and nodes required", http.StatusBadRequest)
		return
	}

	requested := args.Pod.Annotations[poolKey]
	allowed := make([]corev1.Node, 0, len(args.Nodes.Items))
	failedReasons := make(extenderv1.FailedNodeReasonsMap)

	for _, node := range args.Nodes.Items {
		if requested != "" && node.Labels[poolKey] != requested {
			failedReasons[node.Name] = "accelerator pool mismatch"
			continue
		}
		allowed = append(allowed, node)
	}

	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(extenderv1.ExtenderFilterResult{
		Nodes:             &corev1.NodeList{Items: allowed},
		FailedNodeReasons: failedReasons,
	}); err != nil {
		log.Printf("encode response: %v", err)
	}
}

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/filter", filter)
	server := &http.Server{
		Addr:              ":8888",
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
	}
	log.Fatal(server.ListenAndServe())
}

Implementation scope: This example demonstrates node filtering using Pod annotations and node labels. It does not query an external entitlement system or enforce authorization. Production entitlement enforcement requires a trusted policy source and admission controls to prevent unauthorized workloads from selecting restricted accelerator pools.

Connect the scheduler

Deploy this service behind the DNS name below, then pass the configuration to a separate kube-scheduler using --config. It is not sufficient to deploy the extender alone.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: custom-scheduler
extenders:
  - urlPrefix: "http://scheduler-extender.kube-system.svc.cluster.local:8888"
    filterVerb: filter
    nodeCacheCapable: false
    ignorable: false
    httpTimeout: 2s

Opt in a Pod with spec.schedulerName: custom-scheduler. This is a configuration fragment, not a complete secondary-scheduler deployment: provide scheduler credentials, RBAC, leader election, and a reachable Service. Prefer authenticated TLS in production. The endpoint is called by the scheduler, not by the Pod.

This handler supports only Nodes (nodeCacheCapable: false), not NodeNames. It lacks request-size limits, metrics, graceful shutdown, and external entitlement checks: it is a runnable demonstration, not production-ready.

For static labels, use node affinity instead. An extender becomes useful when policy truly requires external decisions.

Prioritization is a preference, not a guarantee

Extenders return node scores from 0 to 10, combined with scheduler scores using an extender weight. Framework plugin scoring uses different conventions.

Scoring might favor lower startup latency or fragmentation, but only if measured with sufficiently fresh data.

Mandatory requirements belong in filtering; preferences belong in scoring.

Preemption requires more care

Preemption can terminate running workloads. An extender must respect scheduler victim-selection semantics, not independently delete Pods.

Application labels and namespace names are not reliable protection boundaries. Use priority classes, preemption policies, and PodDisruptionBudgets where applicable.

Avoid custom preemption in an initial implementation.

Extenders versus Scheduling Framework plugins

Framework plugins provide extension points including PreFilter, Filter, Score, Reserve, Permit, and PreBind, but require maintaining a custom scheduler binary.

DimensionScheduler extenderScheduling Framework plugin
ExecutionExternal HTTP(S) serviceIn-process scheduler plugin
Integration depthFilter, prioritize, and selected additional operationsMultiple scheduling and binding-cycle extension points
LatencyNetwork round trip and service processingNo extender HTTP hop
DeploymentService plus scheduler configurationCustom scheduler build and deployment
Failure domainExternal service availability affects schedulingPlugin behavior affects scheduler process
Best fitExisting extender integration or bounded external decisionDeep lifecycle integration and advanced scheduling behavior

Managed services may prohibit changing the default scheduler. A secondary scheduler requires RBAC, leader election, compatible configuration, and spec.schedulerName on participating Pods.

Failure semantics are part of the architecture

Extender calls are synchronous. Timeouts and ignorable control failure behavior; filter timeouts can block scheduling, while prioritize timeouts may be ignored. See the version-specific scheduler configuration API.

Fail closed for mandatory entitlements, accepting Pending Pods during outages. Fail open only for advisory decisions; an ignorable extender is not a security boundary.

Set latency objectives, bound requests, cache safely, use replicas and network isolation, and monitor errors, timeouts, and outcomes without high-cardinality Pod labels.

For TLS, configure trusted CAs and server identity; client certificate fields serve a different purpose. Request-only filters may need no API-server RBAC.

A practical validation plan

Test on a disposable cluster with a separate scheduler and pinned compatible versions.

  1. Verify the handler’s behavior with allowed, rejected, missing, and malformed inputs.
  2. Verify that the scheduler sends the expected node-list representation. If enabling nodeCacheCapable, implement and test the NodeNames representation separately.
  3. Confirm that a Pod using the custom scheduler lands only on eligible nodes.
  4. Simulate extender timeouts and outages; verify the configured fail-open or fail-closed behavior.
  5. Compare scheduling latency and throughput against a baseline without the extender.
  6. Validate upgrades and rollback before deploying to production.

Unit tests alone cannot prove scheduler integration or outage behavior.

Choosing the right extension point

Prefer native constraints. Consider extenders for narrow external decisions or existing integrations, and framework plugins for deeper lifecycle coordination.

Before coding, define the policy source of truth, freshness guarantees, and outage behavior.

At ape factory, we treat scheduling customization as platform engineering: minimize new control-plane dependencies and test failure modes.

Further reading

Related Articles

Stay ahead

Get the latest insights.

Insights on cloud, AI and open source — delivered to your inbox.

Email subscriptions are coming soon.