Configure the proxy cache

This guide explains how to configure and use proxy cache projects in the Numspot Container Registry.

---

Overview

The proxy cache allows the Numspot Container Registry to act as a pull-through cache for images coming from external registries. When a requested image does not exist locally, Harbor automatically pulls it from the configured upstream registry and caches it for later use.

Benefits

• Reduced latency: Pull images from a local cache instead of external registries
• Bandwidth savings: Reduce external network traffic and the associated costs
• High availability: Access cached images even when the upstream registry is unavailable
• Rate-limit avoidance: Avoid Docker Hub rate limits by caching images locally
• Air-gapped environments: Provide access to external images in restricted networks

Supported upstream registries

The Numspot Container Registry can proxy cache from:

• Docker Hub
• AWS Elastic Container Registry (ECR)
• Azure Container Registry (ACR)
• Google Container Registry (GCR) / Artifact Registry
• Quay
• Other Harbor instances

---

How the proxy cache works

┌─────────────┐        ┌──────────────────────────┐        ┌─────────────────┐
│   Docker    │        │  Numspot Container       │        │   Upstream      │
│   Client    │───────▶│  Registry (Proxy Cache)  │───────▶│   Registry      │
└─────────────┘        └──────────────────────────┘        └─────────────────┘
                                │
                                │ On the first pull:
                                │ 1. Check the local cache
                                │ 2. If not found, pull from upstream
                                │ 3. Cache the image
                                │ 4. Return to the client
                                │
                                │ On subsequent pulls:
                                │ 1. Return the cached image
                                │ 2. Update the cache if a newer version exists

---

Prerequisites

• Registry instance: An operational Numspot Container Registry
• Administrator credentials: Access to the Harbor interface with administrator privileges
• Upstream registry credentials: (Optional) For private repositories
• Network connectivity: The registry must be able to reach the upstream registry

---

Step 1: Create a replication endpoint

Before creating a proxy cache project, configure an endpoint for the upstream registry.

Through the Harbor interface

1. Log in to the Harbor interface
2. Navigate to Administration → Registries
3. Click + New Endpoint
4. Configure the endpoint:

Field	Value
Provider	Docker Hub, AWS ECR, etc.
Name	docker-hub-cache
Endpoint URL	https://registry-1.docker.io (Docker Hub)
Access ID	Your Docker Hub username (optional for public repositories)
Access Secret	Your Docker Hub password/token (optional for public repositories)
Verify Remote Cert	Enabled

5. Click Test Connection
6. Click OK to save

Endpoint configuration examples

Docker Hub (public images)

{
  "name": "docker-hub-public",
  "type": "docker-hub",
  "url": "https://registry-1.docker.io"
}

Docker Hub (private images)

{
  "name": "docker-hub-private",
  "type": "docker-hub",
  "url": "https://registry-1.docker.io",
  "credential": {
    "access_key": "dockerhub-username",
    "access_secret": "dockerhub-access-token"
  }
}

AWS ECR

{
  "name": "aws-ecr",
  "type": "aws-ecr",
  "url": "https://123456789.dkr.ecr.us-east-1.amazonaws.com",
  "credential": {
    "access_key": "AKIAIOSFODNN7EXAMPLE",
    "access_secret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
  }
}

Google GCR

{
  "name": "google-gcr",
  "type": "google-gcr",
  "url": "https://gcr.io/my-project",
  "credential": {
    "access_key": "_json_key",
    "access_secret": "{\"type\":\"service_account\",\"project_id\":\"my-project\",...}"
  }
}

Through the API

curl -X POST "https://registry-{registryId}.hcp.cloudgouv-eu-west-1.numspot.com/api/v2.0/registries" \
  -u "admin:{password}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "docker-hub-cache",
    "type": "docker-hub",
    "url": "https://registry-1.docker.io",
    "credential": {
      "type": "basic",
      "access_key": "dockerhub-username",
      "access_secret": "dockerhub-token"
    }
  }'

---

Step 2: Create a proxy cache project

Through the Harbor interface

1. Log in to the Harbor interface
2. Click Projects → New Project
3. Configure the project:

Field	Value
Name	docker-hub-cache (or a descriptive name)
Access Level	Public (to allow pull for everyone) or Private
Proxy Cache	Enable
Endpoint	Select the docker-hub-cache endpoint created in step 1

4. Click OK to create

Through the API

curl -X POST "https://registry-{registryId}.hcp.cloudgouv-eu-west-1.numspot.com/api/v2.0/projects" \
  -u "admin:{password}" \
  -H "Content-Type: application/json" \
  -d '{
    "project_name": "docker-hub-cache",
    "public": true,
    "registry_id": 1,
    "metadata": {
      "public": "true"
    }
  }'

---

Step 3: Use the proxy cache project

Pull images

Pull images through the proxy cache project:

# Pull an official Docker Hub image through the cache
docker pull registry-{registryId}.hcp.cloudgouv-eu-west-1.numspot.com/docker-hub-cache/library/nginx:latest

# Pull an image from a user repository through the cache
docker pull registry-{registryId}.hcp.cloudgouv-eu-west-1.numspot.com/docker-hub-cache/myuser/myimage:v1.0

# Pull an AWS ECR image through the cache
docker pull registry-{registryId}.hcp.cloudgouv-eu-west-1.numspot.com/aws-ecr-cache/my-app:latest

In Kubernetes

Configure Kubernetes to pull from the proxy cache:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: registry-{registryId}.hcp.cloudgouv-eu-west-1.numspot.com/docker-hub-cache/library/nginx:latest

In Docker Compose

version: '3'
services:
  nginx:
    image: registry-{registryId}.hcp.cloudgouv-eu-west-1.numspot.com/docker-hub-cache/library/nginx:latest

---

Step 4: Configure image pull-through

Automatic caching

When you pull an image through a proxy cache project:

1. Harbor checks whether the image exists locally
2. If it is not found:
   • Harbor pulls the image from the upstream registry
   • Stores it in the proxy cache project
   • Returns it to the client
3. Subsequent pulls serve the cached image

Cache behavior

Scenario	Behavior
First pull	Pull from upstream, cache locally
Subsequent pulls	Serve from the cache
New tag upstream	Pull and cache the new version
Image deleted upstream	Keep serving the cached version
Upstream unavailable	Serve the cached version if available

---

Advanced configuration

Cache multiple registries

Create separate proxy cache projects for different upstream registries:

Project Name	Upstream Registry	Use case
docker-hub-cache	Docker Hub	Official images, Docker Hub repositories
aws-ecr-cache	AWS ECR	Private AWS images
gcr-cache	Google GCR	Google Cloud images
quay-cache	Quay.io	Quay repositories

Cascading cache (multi-tier)

Configure Harbor-to-Harbor proxy cache:

External Registry → Harbor Instance A (Prod) → Harbor Instance B (DR)

Create endpoints pointing to other Numspot registries:

curl -X POST "https://registry-{dest-id}.hcp.cloudgouv-eu-west-1.numspot.com/api/v2.0/registries" \
  -u "admin:{password}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "harbor-prod",
    "type": "harbor",
    "url": "https://registry-{source-id}.hcp.cloudgouv-eu-west-1.numspot.com",
    "credential": {
      "access_key": "admin",
      "access_secret": "source-registry-password"
    }
  }'

---

Monitoring the proxy cache

View cached images

1. Navigate to the proxy cache project in the Harbor interface
2. Browse the repositories to see the cached images
3. Check the pull count and the last pull date

Through the API

# List the repositories in the proxy cache project
curl -X GET "https://registry-{registryId}.hcp.cloudgouv-eu-west-1.numspot.com/api/v2.0/projects/docker-hub-cache/repositories" \
  -u "admin:{password}"

# Get the details of a repository
curl -X GET "https://registry-{registryId}.hcp.cloudgouv-eu-west-1.numspot.com/api/v2.0/projects/docker-hub-cache/repositories/library%2Fnginx" \
  -u "admin:{password}"

Cache statistics

Monitor cache performance:

• Hit rate: Percentage of pulls served from the cache
• Storage usage: Space consumed by the cached images
• Pull frequency: Most frequently pulled images
• Upstream latency: Time to pull from upstream on a cache miss

---

Cache maintenance

Manual cache refresh

Force a refresh of the cached images:

1. Navigate to the proxy cache project
2. Select the repository
3. Click Delete to remove the cached image
4. Pull the image again to refresh it from upstream

Through the API

# Delete a cached repository
curl -X DELETE "https://registry-{registryId}.hcp.cloudgouv-eu-west-1.numspot.com/api/v2.0/projects/docker-hub-cache/repositories/library%2Fnginx" \
  -u "admin:{password}"

Automatic cache cleanup

Configure retention policies:

1. Navigate to Administration → Configuration → System Settings
2. Set Artifact Read-only for specific projects
3. Configure automatic cleanup (see Automatic cleanup)

---

Best practices

Performance

1. Geographic proximity: Create proxy cache projects in regions close to your workloads
2. Cache pre-warming: Pull frequently used images before deployment to production
3. Use public projects: Make proxy cache projects public to avoid authentication overhead
4. Monitor usage: Track cache hit rates and adjust the strategy accordingly

Security

1. Validate upstream images: Enable vulnerability scanning for cached images
2. Use signed images: Configure trust policies for upstream registries
3. Restrict access: Use private proxy cache projects for sensitive images
4. Audit pulls: Review the audit logs for proxy cache access

Operations

1. Set storage quotas: Configure quotas on proxy cache projects to prevent unbounded growth
2. Regular cleanup: Implement automatic cleanup policies
3. Monitor upstream: Monitor upstream registry availability and update the endpoints if necessary
4. Document the mappings: Maintain documentation of the mappings between proxy cache projects and upstream registries

---

Troubleshooting

Issue 1: image not cached

Symptom: Pulls are slow, images do not appear in the proxy cache project

Solutions:

1. Check that the proxy cache project is configured correctly
2. Check the endpoint configuration and the credentials
3. Check network connectivity to the upstream registry
4. Review the Harbor logs:

kubectl logs -n registry-instance-{registryId} deployment/harbor-core

Issue 2: authentication failure

Symptom: Unable to pull private images through the proxy cache

Solutions:

1. Check that the endpoint credentials are correct
2. Check that the credentials have pull permissions on the upstream registry
3. Update the endpoint credentials if they have expired
4. Test the endpoint connection in the Harbor interface

Issue 3: stale cached images

Symptom: The cached images are out of date

Solutions:

1. Delete the cached repository to force a refresh
2. Pull with a specific digest instead of a tag
3. Configure cache invalidation policies
4. Use the --no-cache flag to bypass the cache

Issue 4: upstream registry rate limits

Symptom: Unable to pull new images, rate-limit errors

Solutions:

1. Use an authenticated endpoint (Docker Hub offers higher limits for authenticated users)
2. Pre-warm the cache during off-peak hours
3. Reduce pull frequency by using stable image tags
4. Consider mirroring critical images instead of using the proxy cache

---

API reference

Create a proxy cache project

POST /api/v2.0/projects
Authorization: Basic {base64(admin:password)}
Content-Type: application/json

{
  "project_name": "string",
  "public": true,
  "registry_id": number,
  "metadata": {
    "public": "true"
  }
}

List proxy cache projects

GET /api/v2.0/projects
Authorization: Basic {base64(admin:password)}

Get project details

GET /api/v2.0/projects/{project_id}
Authorization: Basic {base64(admin:password)}

List cached repositories

GET /api/v2.0/projects/{project_name}/repositories
Authorization: Basic {base64(admin:password)}

---

Limitations

• The proxy cache is read-only: You cannot push images to proxy cache projects
• Images are cached on the first pull: No automatic pre-fetching
• The cache is tag-based: Using digests bypasses cache validation
• Storage is unbounded by default: Configure quotas to limit growth
• Upstream availability: Cache misses require connectivity to the upstream registry

---