Building Custom Kubernetes Scheduling: Extenders vs. Plugins
Understand when Kubernetes scheduler extenders make sense, how they affect the scheduling path, and when the Scheduling Framework is the better choice.

On this page
Kubernetes schedules Pods through feasibility checks, scoring, and binding. Native node affinity, topology spread constraints, taints, tolerations, and resource requests cover most placement needs.
Some decisions depend on external information: accelerator licenses, for example, or rapidly changing infrastructure policies.
Two approaches are scheduler extenders (external HTTP services) and Scheduling Framework plugins (in-process extensions).
Start with the scheduling primitives you already have
First exhaust declarative options: node affinity for GPU models, device plugins and extended resources for hardware capacity, and topology spread constraints for failure domains.
If a policy depends on external state, consider synchronizing that state into Kubernetes before adding synchronous scheduling calls.
Where an extender fits
A configured extender is an HTTP(S) service that can filter or score candidate nodes and optionally participate in preemption or binding.
The diagram could not be displayed. Its source is available below.
Diagram source
flowchart TD
A[Pending Pod] --> B[Scheduler filters candidate nodes]
B --> C[Extender filter endpoint]
C --> D{Feasible nodes remain?}
D -->|Yes| E[Scheduler scores nodes]
E --> F[Extender prioritize endpoint]
F --> G[Select node and bind Pod]
D -->|No| H[Unschedulable handling and possible preemption]This simplifies the scheduler’s actual cycles. An extender is not an admission webhook; preemption calls occur only under specific conditions.
Extenders avoid custom scheduler binaries but add a synchronous network dependency.
A concrete use case: accelerator entitlement
Consider GPU nodes whose use depends on accelerator-pool entitlement. The scheduler sees GPU capacity but not the external entitlement decision.
For illustration, compare the Pod annotation scheduling.apefactory.com/pool with a trusted node label of the same name. This does not verify external entitlements.
package main
import (
"encoding/json"
"log"
"net/http"
"time"
corev1 "k8s.io/api/core/v1"
extenderv1 "k8s.io/kube-scheduler/extender/v1"
)
const poolKey = "scheduling.apefactory.com/pool"
func filter(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
var args extenderv1.ExtenderArgs
if err := json.NewDecoder(r.Body).Decode(&args); err != nil {
http.Error(w, "invalid request", http.StatusBadRequest)
return
}
if args.Pod == nil || args.Nodes == nil {
http.Error(w, "pod and nodes required", http.StatusBadRequest)
return
}
requested := args.Pod.Annotations[poolKey]
allowed := make([]corev1.Node, 0, len(args.Nodes.Items))
failedReasons := make(extenderv1.FailedNodeReasonsMap)
for _, node := range args.Nodes.Items {
if requested != "" && node.Labels[poolKey] != requested {
failedReasons[node.Name] = "accelerator pool mismatch"
continue
}
allowed = append(allowed, node)
}
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(extenderv1.ExtenderFilterResult{
Nodes: &corev1.NodeList{Items: allowed},
FailedNodeReasons: failedReasons,
}); err != nil {
log.Printf("encode response: %v", err)
}
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/filter", filter)
server := &http.Server{
Addr: ":8888",
Handler: mux,
ReadHeaderTimeout: 5 * time.Second,
}
log.Fatal(server.ListenAndServe())
}Implementation scope: This example demonstrates node filtering using Pod annotations and node labels. It does not query an external entitlement system or enforce authorization. Production entitlement enforcement requires a trusted policy source and admission controls to prevent unauthorized workloads from selecting restricted accelerator pools.
Connect the scheduler
Deploy this service behind the DNS name below, then pass the configuration to a separate kube-scheduler using --config. It is not sufficient to deploy the extender alone.
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: custom-scheduler
extenders:
- urlPrefix: "http://scheduler-extender.kube-system.svc.cluster.local:8888"
filterVerb: filter
nodeCacheCapable: false
ignorable: false
httpTimeout: 2sOpt in a Pod with spec.schedulerName: custom-scheduler. This is a configuration fragment, not a complete secondary-scheduler deployment: provide scheduler credentials, RBAC, leader election, and a reachable Service. Prefer authenticated TLS in production. The endpoint is called by the scheduler, not by the Pod.
This handler supports only Nodes (nodeCacheCapable: false), not NodeNames. It lacks request-size limits, metrics, graceful shutdown, and external entitlement checks: it is a runnable demonstration, not production-ready.
For static labels, use node affinity instead. An extender becomes useful when policy truly requires external decisions.
Prioritization is a preference, not a guarantee
Extenders return node scores from 0 to 10, combined with scheduler scores using an extender weight. Framework plugin scoring uses different conventions.
Scoring might favor lower startup latency or fragmentation, but only if measured with sufficiently fresh data.
Mandatory requirements belong in filtering; preferences belong in scoring.
Preemption requires more care
Preemption can terminate running workloads. An extender must respect scheduler victim-selection semantics, not independently delete Pods.
Application labels and namespace names are not reliable protection boundaries. Use priority classes, preemption policies, and PodDisruptionBudgets where applicable.
Avoid custom preemption in an initial implementation.
Extenders versus Scheduling Framework plugins
Framework plugins provide extension points including PreFilter, Filter, Score, Reserve, Permit, and PreBind, but require maintaining a custom scheduler binary.
| Dimension | Scheduler extender | Scheduling Framework plugin |
|---|---|---|
| Execution | External HTTP(S) service | In-process scheduler plugin |
| Integration depth | Filter, prioritize, and selected additional operations | Multiple scheduling and binding-cycle extension points |
| Latency | Network round trip and service processing | No extender HTTP hop |
| Deployment | Service plus scheduler configuration | Custom scheduler build and deployment |
| Failure domain | External service availability affects scheduling | Plugin behavior affects scheduler process |
| Best fit | Existing extender integration or bounded external decision | Deep lifecycle integration and advanced scheduling behavior |
Managed services may prohibit changing the default scheduler. A secondary scheduler requires RBAC, leader election, compatible configuration, and spec.schedulerName on participating Pods.
Failure semantics are part of the architecture
Extender calls are synchronous. Timeouts and ignorable control failure behavior; filter timeouts can block scheduling, while prioritize timeouts may be ignored. See the version-specific scheduler configuration API.
Fail closed for mandatory entitlements, accepting Pending Pods during outages. Fail open only for advisory decisions; an ignorable extender is not a security boundary.
Set latency objectives, bound requests, cache safely, use replicas and network isolation, and monitor errors, timeouts, and outcomes without high-cardinality Pod labels.
For TLS, configure trusted CAs and server identity; client certificate fields serve a different purpose. Request-only filters may need no API-server RBAC.
A practical validation plan
Test on a disposable cluster with a separate scheduler and pinned compatible versions.
- Verify the handler’s behavior with allowed, rejected, missing, and malformed inputs.
- Verify that the scheduler sends the expected node-list representation. If enabling nodeCacheCapable, implement and test the NodeNames representation separately.
- Confirm that a Pod using the custom scheduler lands only on eligible nodes.
- Simulate extender timeouts and outages; verify the configured fail-open or fail-closed behavior.
- Compare scheduling latency and throughput against a baseline without the extender.
- Validate upgrades and rollback before deploying to production.
Unit tests alone cannot prove scheduler integration or outage behavior.
Choosing the right extension point
Prefer native constraints. Consider extenders for narrow external decisions or existing integrations, and framework plugins for deeper lifecycle coordination.
Before coding, define the policy source of truth, freshness guarantees, and outage behavior.
At ape factory, we treat scheduling customization as platform engineering: minimize new control-plane dependencies and test failure modes.


