Configure NTP in an airgap environment

In an airgap environment (without direct Internet access), time synchronization between the nodes of a cluster Kubernetes is entirely delegated to the customer. Configure NTP (Network Time Protocol) on the Numspot workers to avoid time synchronization issues.

info

Prerequisites

• Have an operational Kubernetes cluster running in private mode (airgap).
• Have retrieved the kubeconfig file.
• Have configured an internal NTP server (for example: a stratum 1 or 2 server internal to the organization).
• Have an internal image registry if you deploy through a DaemonSet.

Why NTP is critical in airgap

A clock skew between the Kubernetes workers can lead to:

• TLS (Transport Layer Security)/SSL (Secure Sockets Layer) connection failures;
• rejection of authentication tokens and encrypted volumes;
• monitoring tool malfunctions (disordered logs, inconsistent metrics);
• loss of scheduler coordination (for example, the lease API).

From installation, Numspot Kubernetes clusters inject a default NTP server (for example, ntp.numspot.local). In airgap, this server must be replaced by an internal time source.

Architecture and responsibilities

Possible approaches

The customer has two approaches to configure NTP:

Approach	Description	Use case
user-data / cloud-init	Configuration at node start-up through the instance metadata.	Automatic when the node pool is created.
Kubernetes DaemonSet	Deployment of an NTP agent through a pod running on each node.	Subsequent modification on an existing cluster.

Time service configuration on the workers

The Numspot workers use systemd-timesyncd or chronyd depending on the ANSSI (French National Cybersecurity Agency) system image:

• systemd-timesyncd: a lightweight service sufficient for most workloads;
• chronyd: recommended if high-precision synchronization is required (for example, transactional databases).

Securing the NTP traffic

In an airgap environment, restrict access to port 123/UDP (User Datagram Protocol) between the workers and the NTP server through a Network Policy or the OS (Operating System) firewall.

Configuration through user-data

When you create a node pool, you can provide a cloud-init script to point immediately to your internal NTP server.

user-data example for systemd-timesyncd

#cloud-config
runcmd:
  - systemctl stop systemd-timesyncd || true
  - sed -i 's/^#*NTP=.*/NTP=ntp.interne.monorg.lan/' /etc/systemd/timesyncd.conf
  - sed -i 's/^#*FallbackNTP=.*/FallbackNTP=ntp2.interne.monorg.lan/' /etc/systemd/timesyncd.conf
  - systemctl restart systemd-timesyncd
  - timedatectl set-ntp true

user-data example for chronyd

#cloud-config
packages:
  - chrony
runcmd:
  - systemctl stop chronyd || true
  - echo "server ntp.interne.monorg.lan iburst" > /etc/chrony/sources.d/local-ntp.sources
  - echo "server ntp2.interne.monorg.lan iburst" >> /etc/chrony/sources.d/local-ntp.sources
  - systemctl restart chronyd

info

These scripts are applied as soon as the nodes are created. For existing nodes, favor the DaemonSet approach described below.

Configuration through a Kubernetes DaemonSet

To modify the NTP configuration afterwards, without recreating the nodes, deploy a privileged Linux DaemonSet that modifies the host configuration.

Step 1: containerize the NTP configuration

Create a simple Dockerfile:

FROM alpine:3.19
RUN apk add --no-cache tzdata systemd
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

The entrypoint.sh script:

#!/bin/sh
set -e

NTP_SERVER=${NTP_SERVER:-ntp.interne.monorg.lan}
FALLBACK_SERVER=${FALLBACK_SERVER:-ntp2.interne.monorg.lan}

if [ -f /host/etc/systemd/timesyncd.conf ]; then
  echo "Configuration de systemd-timesyncd..."
  sed -i "s/^#*NTP=.*/NTP=${NTP_SERVER}/" /host/etc/systemd/timesyncd.conf
  sed -i "s/^#*FallbackNTP=.*/FallbackNTP=${FALLBACK_SERVER}/" /host/etc/systemd/timesyncd.conf
  chroot /host systemctl restart systemd-timesyncd || true
fi

if [ -f /host/etc/chrony/chrony.conf ] || [ -f /host/etc/chrony.conf ]; then
  echo "Configuration de chronyd..."
  CONF_FILE="/host/etc/chrony/chrony.conf"
  [ -f /host/etc/chrony.conf ] && CONF_FILE="/host/etc/chrony.conf"
  sed -i '/^server /d' "${CONF_FILE}"
  echo "server ${NTP_SERVER} iburst" >> "${CONF_FILE}"
  echo "server ${FALLBACK_SERVER} iburst" >> "${CONF_FILE}"
  chroot /host systemctl restart chronyd || true
fi

echo "Time synchronization configured."
sleep infinity

Step 2: prepare the Kubernetes manifest

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ntp-config
  namespace: kube-system
  labels:
    app: ntp-config
spec:
  selector:
    matchLabels:
      app: ntp-config
  template:
    metadata:
      labels:
        app: ntp-config
    spec:
      hostNetwork: true
      hostPID: true
      containers:
        - name: ntp-config
          image: <REGISTRE_INTERNE>/ntp-config:latest
          imagePullPolicy: Always
          env:
            - name: NTP_SERVER
              value: "ntp.interne.monorg.lan"
            - name: FALLBACK_SERVER
              value: "ntp2.interne.monorg.lan"
          volumeMounts:
            - name: host
              mountPath: /host
          securityContext:
            privileged: true
            runAsUser: 0
      volumes:
        - name: host
          hostPath:
            path: /
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule

Replace <REGISTRE_INTERNE> with the address of your airgap registry (for example, registry.intranet.monorg.lan:5000).

Elevated privileges

This DaemonSet runs in privileged mode. Adjusting the security context is the responsibility of the customer. Remove the DaemonSet once the configuration has been applied to limit the attack surface.

Step 3: deploy and verify

kubectl apply -f ntp-config-daemonset.yaml
kubectl rollout status ds/ntp-config -n kube-system

Verifying the synchronization

Check the state on the nodes

kubectl get nodes -o wide

Identify a node and run:

NODE=<NODE_NAME>
kubectl debug node/${NODE} -it --image=busybox:stable -- /bin/sh

In the debug container:

chroot /host

## Pour systemd-timesyncd
timedatectl status
systemctl status systemd-timesyncd

## Pour chronyd
chronyc tracking
chronyc sources

Expected output (systemd-timesyncd example)

         Universal time: jeu. 2025-09-04 14:30:00 UTC
       NTP synchronized: yes
NTP enabled: yes

Expected output (chronyd example)

Reference ID    : 192.168.56.10 (ntp.interne.monorg.lan)
Stratum         : 2
Last offset     : +0.000001234 seconds
RMS offset      : 0.000002345 seconds

Check the clock skew in Kubernetes

Run an ephemeral pod to compare the dates between the control plane and the workers:

kubectl run date-check --image=busybox:stable --rm -it --restart=Never -- date

The difference between the displayed date and that of your local machine must be less than a few seconds.

Prometheus metric

If you have an observability stack inside the airgap, monitor the node_timex_sync_status metric (node_exporter). A value of 1 indicates that the node is synchronized.

Troubleshooting

The NTP service does not start

Cause: the node has no network connectivity with the NTP server (port 123/UDP blocked).

Solution: explicitly allow the UDP/123 traffic between the workers and the internal NTP server; test from a node:

chronyd -Q 'server ntp.interne.monorg.lan iburst'

NTP synchronized: no error despite the configuration

Cause: the systemd-timesyncd service conflicts with chronyd.

Solution: disable the unused service:

systemctl disable --now chronyd   # if you use timesyncd
systemctl disable --now systemd-timesyncd  # if you use chronyd

The pods do not start after a time change

Cause: an abrupt time jump caused the expiry of certificates or serviceAccount tokens.

Solution:

1. Check the validity of the cluster certificates:

   openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates

2. Restart the affected pods to regenerate the tokens:

   kubectl rollout restart deployment/<name> -n <namespace>

3. In case of a large gap (several minutes), renew the certificates with kubeadm certs renew all on the control planes (support required).

Best practices

Best practice	Detail
Redundant NTP server	Define at least two internal NTP sources (primary and fallback).
Network isolation	Restrict port 123/UDP only towards your authorized NTP servers.
Continuous monitoring	Monitor node_timex_sync_status and raise an alert if a node loses synchronization.
Non-regression	Manage the NTP user-data through your infrastructure as code chain (Terraform, Pulumi) to ensure consistency on each node pool creation.

Integration into the node pool creation cycle

When you create a node pool, systematically include the NTP user-data script in your automation. Example with the Numspot API:

{
  "name": "pool-prod-01",
  "nodeType": "standard-4",
  "minNodes": 3,
  "maxNodes": 10,
  "userData": "IyEvYmluL2Jhc2gKc3VkbyBzeXN0ZW1jdGwgc3RvcCBzeXN0ZW1kLXRpbWVzeW5jZCA..."
}

The userData field must contain your script encoded in base64, ensuring that each node joins the cluster with the correct time configuration.

References

• Kubernetes documentation – Node Problem Detector ↗
• systemd-timesyncd ↗
• chronyc ↗