Expose an application internally through a private load balancer

By default, the applications of a Kubernetes cluster are exposed to the Internet through the public ingress managed by the platform. When the monitoring or alerting tools are hosted in another VPC (Virtual Private Cloud) — or on-premises through a network peering — it is preferable to reach them through a private endpoint, without public exposure.

This guide describes the creation of a LoadBalancer type Service: the platform provisions a private load balancer that allocates private IP (Internet Protocol) addresses to the application, reachable from the networks peered with the cluster network.

Prerequisites

info

• Have an operational Numspot Kubernetes cluster ↗
• Download the kubeconfig file ↗
• kubectl installed and configured
• An active peering between the network of the monitoring tools and the cluster network
• A dedicated security group for the load balancer ↗

Public and internal load balancer

By default, a LoadBalancer type Service provisions a public load balancer: the platform allocates public IP addresses to it and exposes it to the Internet.

The service.beta.kubernetes.io/osc-load-balancer-internal: "true" annotation creates an internal load balancer: it receives private IP addresses in the specified subnet and is only reachable from the cluster network and the peered networks. The platform itself relies on this same mechanism for its own internal components.

IP address lifecycle

IP address type	Allocation	Stability
Private IP (internal)	Allocated automatically in the subnet at creation time	Stable for the whole life of the load balancer, not pinnable, changed if the Service is recreated
Pre-allocated static IP	IP address of the account referenced by annotation before creation	Keeps the same IP address even if the load balancer is recreated

Firewall rules must not target a private IP address hardcoded: prefer filtering by security group — an inbound rule allowing the load balancer security group — or by CIDR (Classless Inter-Domain Routing) — the ranges of the peered networks.

For a private endpoint with a stable reference on the client side, use a private DNS (Domain Name System) name: a CNAME record isolates the IP address value, and recreating the load balancer only requires updating a single DNS record.

Procedure

note

The annotation keys used in this guide are defined by the cloud controller manager of the platform: use them as they are, respecting the case. Only use the keys documented on this page.

Step 1: Check the network peering

Check that the VPC of the monitoring tools is peered with the cluster network and that the route table of both networks allows traffic between the relevant CIDR ranges.

Step 2: Prepare the subnet and the security group

1. Identify the subnet of the cluster network in which the load balancer will be created: the private IP addresses are allocated in this subnet. Retrieve its identifier from the console or the network API.
2. Create a security group dedicated to the load balancer and add an inbound rule allowing the CIDR of the peered networks towards the listening port of the Service. See Create a security group ↗ and Add a rule to a security group ↗.
3. If traffic is filtered between the load balancer and the nodes of the cluster, add a rule allowing the load balancer security group to reach the target port of the application.

Step 3: Declare the Service

Create the namespace if needed, apply the manifest described in the Example manifests section then check the result:

kubectl create namespace demo
kubectl apply -f internal-service.yaml

Main annotations

Annotation	Example value	Description
service.beta.kubernetes.io/osc-load-balancer-internal	"true"	Creates an internal load balancer: private IP addresses in the specified subnet
service.beta.kubernetes.io/osc-load-balancer-subnet-id	"11111111-2222-3333-4444-…"	UUID (Universally Unique Identifier) of the subnet in which the load balancer is created
service.beta.kubernetes.io/osc-load-balancer-name	"lb-demo-internal"	Fixed name of the load balancer, 32 characters maximum
service.beta.kubernetes.io/osc-load-balancer-security-group	"aaaaaaaa-bbbb-cccc-dddd-…"	Main security group of the load balancer
service.beta.kubernetes.io/osc-load-balancer-extra-security-groups	"11111111-…, 22222222-…"	Additional security groups, separated by commas
service.beta.kubernetes.io/load-balancer-source-ranges	"198.51.100.0/24, 203.0.113.0/24"	CIDR sources allowed to reach the load balancer, without additional prefix

Additional annotations

Annotation	Example value	Description
service.beta.kubernetes.io/osc-load-balancer-healthcheck-protocol	"HTTP"	Health check protocol: HTTP or TCP
service.beta.kubernetes.io/osc-load-balancer-healthcheck-port	"8080"	Port queried by the health check
service.beta.kubernetes.io/osc-load-balancer-healthcheck-path	"/healthz"	Path queried, for an HTTP health check
service.beta.kubernetes.io/osc-load-balancer-healthcheck-interval	"30"	Interval between two health checks, in seconds
service.beta.kubernetes.io/osc-load-balancer-healthcheck-timeout	"5"	Timeout of each health check, in seconds
service.beta.kubernetes.io/osc-load-balancer-healthcheck-healthy-threshold	"2"	Number of consecutive successes before declaring the target healthy
service.beta.kubernetes.io/osc-load-balancer-healthcheck-unhealthy-threshold	"3"	Number of consecutive failures before removing the target
service.beta.kubernetes.io/osc-load-balancer-ip-id	"ffffffff-…"	Pre-allocated public IP address of the account, attached to the load balancer
service.beta.kubernetes.io/osc-load-balancer-ip-pool	"my-pool-prod"	Selects an IP address of the account carrying the OscK8sIPPool:<pool-name> tag
service.beta.kubernetes.io/osc-load-balancer-ingress-address	"ip"	Address exposed by the Service: ip, hostname or both
service.beta.kubernetes.io/osc-load-balancer-ingress-ipmode	"Proxy"	Target addressing mode: Proxy or VIP

Example manifests

Complete internal load balancer

apiVersion: v1
kind: Service
metadata:
  name: demo-internal
  namespace: demo
  annotations:
    # Internal load balancer: private IP addresses in the specified subnet
    service.beta.kubernetes.io/osc-load-balancer-internal: "true"
    service.beta.kubernetes.io/osc-load-balancer-subnet-id: "11111111-2222-3333-4444-555555555555"
    service.beta.kubernetes.io/osc-load-balancer-name: "lb-demo-internal"
    service.beta.kubernetes.io/osc-load-balancer-security-group: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
    # CIDR sources allowed to reach the load balancer
    service.beta.kubernetes.io/load-balancer-source-ranges: "198.51.100.0/24, 203.0.113.0/24"
    # Health check adjusted for the application
    service.beta.kubernetes.io/osc-load-balancer-healthcheck-protocol: "HTTP"
    service.beta.kubernetes.io/osc-load-balancer-healthcheck-port: "8080"
    service.beta.kubernetes.io/osc-load-balancer-healthcheck-path: "/healthz"
    service.beta.kubernetes.io/osc-load-balancer-healthcheck-interval: "30"
    service.beta.kubernetes.io/osc-load-balancer-healthcheck-timeout: "5"
    service.beta.kubernetes.io/osc-load-balancer-healthcheck-healthy-threshold: "2"
    service.beta.kubernetes.io/osc-load-balancer-healthcheck-unhealthy-threshold: "3"
spec:
  type: LoadBalancer
  selector:
    app: demo
  ports:
    - name: http
      port: 8080
      targetPort: 8080
      protocol: TCP

Variant: pre-allocated static IP

This variant answers the need for a fixed IP address for firewall rules: the pre-allocated IP address of the account is kept even if the load balancer is recreated. The exposure remains public however: apply a strict filtering with load-balancer-source-ranges and with security groups.

apiVersion: v1
kind: Service
metadata:
  name: demo-static
  namespace: demo
  annotations:
    # Fixed name and pre-allocated IP: the value survives the recreation of the load balancer
    service.beta.kubernetes.io/osc-load-balancer-name: "lb-demo-static"
    service.beta.kubernetes.io/osc-load-balancer-ip-id: "ffffffff-8888-9999-aaaa-bbbbbbbbbbbb"
spec:
  type: LoadBalancer
  selector:
    app: demo
  ports:
    - name: https
      port: 443
      targetPort: 8443
      protocol: TCP

The IP address must be pre-allocated before the creation of the Service: see Allocate a public IP ↗. Instead of a single address, the osc-load-balancer-ip-pool annotation automatically selects an IP address of the account carrying the OscK8sIPPool:<pool-name> tag.

Verification

Retrieve the load balancer address

kubectl get service demo-internal -n demo

During the creation of the load balancer, the EXTERNAL-IP column displays <pending> then the allocated private IP address:

NAME             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
demo-internal    LoadBalancer   10.100.0.42    192.0.2.17    8080:30691/TCP   2m

To check the applied annotations and the creation events:

kubectl describe service demo-internal -n demo

Check the load balancer targets

The endpoints of the Service must reference the IP addresses of the application pods:

kubectl get endpoints demo-internal -n demo

Test the access from a peered network

From a VM (Virtual Machine) of the remote VPC, or from the on-premises network through the peering:

curl http://192.0.2.17:8080/healthz

An HTTP 200 response confirms that the application is reachable through the load balancer. If the Service is exposed through a hostname (osc-load-balancer-ingress-address), test the DNS resolution from the peered network.

Limitations and best practices

• Create a dedicated LoadBalancer type Service for each internal exposure: do not modify the Service managed by the platform for the public ingress of the cluster.
• The private IP addresses of the internal load balancer are stable during its whole life, but are not pinnable: they change if the Service is deleted then recreated.
• Do not target a private IP address hardcoded in your firewall rules: target security groups or CIDR ranges.
• Reference the IP address through a CNAME record in a private DNS zone rather than communicating it to the client configurations.
• Set the load balancer name with the osc-load-balancer-name annotation to guarantee reproducibility with your automation tools.
• Always declare load-balancer-source-ranges to restrict the access to the allowed networks only.

warning

The private IP address of an internal load balancer changes if the Service is deleted then recreated. A firewall rule targeting this address hardcoded will silently become obsolete: use a filtering by security group or by CIDR range.

Troubleshooting

The Service stays pending

Cause: an annotation is invalid — non-existing subnet, load balancer name longer than 32 characters, IP address not pre-allocated or already attached to another resource.

Solution: check the events with kubectl describe service demo-internal -n demo, fix the relevant annotation then recreate the Service.

Connection impossible from the remote VPC

Cause: the peering, the routing or the filtering blocks the traffic between the remote network and the load balancer.

Solution: check in order:

1. the peering between the two networks and the associated route tables;
2. the inbound rule of the load balancer security group (listening port and source CIDR);
3. the value of load-balancer-source-ranges, which must include the CIDR of the source network;
4. the state of the targets with kubectl get endpoints demo-internal -n demo: empty endpoints indicate a failing health check.

The health check fails

Cause: the osc-load-balancer-healthcheck-* annotations do not match the application — wrong protocol, port or path.

Solution: align the health check annotations with the application, for example HTTP, port 8080 and path /healthz, then check that the endpoints fill in again.

The IP address has changed

Cause: the Service has been deleted then recreated — expected behaviour for a private IP address, which is not pinnable.

Solution: use a CNAME record in a private DNS zone and automate the address update. For a truly fixed address, use the pre-allocated static IP variant.