Skip to content

System Configuration

Nameserver in /etc/resolv.conf

CoreDNS requires an upstream DNS server to forward requests to.

Make sure that the node has at least one nameserver defined in the /etc/resolv.conf file else, the core DNS pods will fail to start with plugin/forward: no nameservers found

cat /etc/resolv.conf 
# DNS requests are forwarded to the host. DHCP DNS options are ignored.
nameserver 192.168.65.7

Reference: CoreDNS CrashLoopBackOff Discussion

Enable IP forwarding

2.1 Check current IP forwarding status

Most systems will be able to use the sysctl command, which can apply kernel variables. Therefore, you can use the following sysctl command to check whether IP forwarding is enabled or disabled.

sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0

sysctl net.ipv4.conf.all.forwarding
net.ipv4.conf.all.forwarding = 0

In the example above, the net.ipv4.ip_forward kernel setting is 0. That means it's off. If it were set to 1, that would mean it's enabled.

2.2 Enable IP forwarding

To make sure the new setting survives a reboot, you need to edit the /etc/sysctl.conf file.

sudo nano /etc/sysctl.conf

Update the net.ipv4.ip_forward and net.ipv4.conf.all.forwarding settings to enable.
If there is no such entry, then add the below line to the bottom of the file.
Save your changes to this file. The setting will be permanent across reboots. The net.ipv4.ip_forward setting controls whether IP forwarding is turned on or off for IPv4.

net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1

After editing the file, you can run the following command to make the changes take effect right away.

sysctl -p

max_user_instances needs to be 8192 (Single node only)

Execute the below command:

sysctl fs.inotify.max_user_instances

If the output of the above command is less that 8192, in that case, you might face too many open files error messages.

In order to avoid these messages, set the fs.inotify.max_user_instances=8192

You need to edit the /etc/sysctl.conf file.

sudo nano /etc/sysctl.conf

Add the below line:

fs.inotify.max_user_instances=8192

After editing the file, you can run the following command to make the changes take effect right away.

sysctl -p

Use iptables instead of ipvs

The on-prem Kubernetes clusters installed with kurl.sh have kube-proxy running in ipvs mode with a default tcp timeout of 900 sec (15 mins), so when there is an idle time of 15 mins or more in a connection to a Kubernetes svc, kube-proxy (ipvs) is dropping it from tracking due to this timeout.

Change the mode from ipvs to iptables and restart kube-proxy.

kubectl edit cm kube-proxy -n kube-system

apiVersion: v1
data:
  config.conf: |-
    mode: iptables

Whitelist domains

For k8s installation through kurl

Whitelist these domains:

References:

Note: Air Gapped install of k8s could be done but if local disk space runs out then images are downloaded over the internet. Hence these domains.

For Traceable platform installation

Whitelist these domains to pull the Traceable related images/artifacts:

If you need to whitelist IPs

Image Pinning in an airgap installation

Kubernetes relies on the kubelet to manage container images on each node. When disk usage hits a configurable threshold (e.g., 85% by default), the kubelet calls the underlying container runtime – often containerd-to prune images that aren't referenced by running containers. This process considers factors like:

  • Disk pressure: High and low thresholds (imageGCHighThresholdPercent and imageGCLowThresholdPercent).
  • Image age: A minimum age before GC kicks in (imageMinimumGCAge).
  • Usage: Only images not tied to active containers are eligible.

In case of installation where the access to internet is blocked, we need to make sure that pause image is not removed due to GC.

Run the below command on each node:

ctr -n k8s.io i ls  | grep k8s | awk '{print $3}'  \
| xargs -I{} bash -c 'ctr -n k8s.io i ls | grep {}'| awk '{print $1}' \
| xargs -I{} ctr -n k8s.io i label {} io.cri-containerd.pinned=pinned