Practical Guide: Debugging Kubernetes Nodes and Managing Docker in Custom Setups

1. Diagnosing Kubernetes Node NotReady Issues

Problem:

Worker nodes node0 and node1 were stuck in NotReady status.

Cause:

  • CNI (Container Network Interface) plugin not initialized.
  • Invalid subnet declaration in 10-bridge.conf (e.g., included hostname/IP instead of CIDR).

Fix:

  1. Install CNI plugins:

    wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
    sudo mkdir -p /opt/cni/bin /etc/cni/net.d
    sudo tar -xzvf cni-plugins-linux-amd64-v1.3.0.tgz -C /opt/cni/bin
    
  2. Create CNI config:

    {
      "cniVersion": "1.0.0",
      "name": "bridge",
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "ranges": [[{"subnet": "10.200.0.0/24"}]],
        "routes": [{"dst": "0.0.0.0/0"}]
      }
    }
    
  3. Restart the kubelet:

    sudo systemctl restart kubelet
    

2. Automating Hostname and SSH Configuration

Goals:

  • Enable root SSH login
  • Set hostnames and update /etc/hosts

Enable Root Login:

sed -i 's/^#*PermitRootLogin no.*/PermitRootLogin yes/' /etc/ssh/sshd_config

Bulk Hostname Setup Script:

while read IP FQDN HOST SUBNET; do
  CLEAN_HOST=$(echo "$HOST" | sed 's/[^a-zA-Z0-9-]//g')
  CMD="sed -i 's/^127.0.1.1.*/127.0.1.1\t${FQDN} ${CLEAN_HOST}/' /etc/hosts"
  ssh -n -p 222 root@$IP "$CMD"
  ssh -n -p 222 root@$IP hostnamectl set-hostname $CLEAN_HOST
  ssh -n -p 222 root@$IP systemctl restart systemd-hostnamed
done < machines.txt

Ensure HOST in machines.txt contains valid characters only.


3. Fixing kubectl Connection Errors

Problem:

kubectl apply -f deploy.yaml
error: failed to download openapi: Get "http://localhost:8080/openapi/v2": connection refused

Diagnosis:

  • No kubeconfig context set.
  • Docker Desktop’s Kubernetes conflicting.

Fix:

  1. Get kubeconfig from service machine:

    scp user@<server-ip>:/etc/kubernetes/admin.conf ~/admin.kubeconfig
    export KUBECONFIG=~/admin.kubeconfig
    
  2. Confirm:

    kubectl get nodes
    

4. Private Docker Registry & Deployment

Set up Private Registry:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

Push Images:

docker tag fano-automation:v1 192.168.3.180:5000/fano-automation:v1
docker push 192.168.3.180:5000/fano-automation:v1

Allow Insecure Registry:

Edit /etc/docker/daemon.json:

{
  "insecure-registries": ["192.168.3.180:5000"]
}

Then restart Docker:

sudo systemctl restart docker

Use in Deployment YAML:

image: 192.168.3.180:5000/fano-automation:v1

5. Accessing NodePort Services

Example:

fano-automation-service   NodePort   8888:32669/TCP

Access the service:

http://192.168.3.182:32669

To expose externally, port-forward in your router to that IP:PORT.


Summary

This guide walks through debugging NotReady nodes, setting up networking and hostnames, resolving kubectl issues, configuring private Docker registries, and deploying services using Kubernetes.

For advanced users: consider setting up TLS for Docker registry, or using ingress with Let’s Encrypt.

Need more? Export to a Markdown wiki or integrate into your internal DevOps handbook!

Recommend learning resource: Kubernetes-the-hard-way , to build a kubernetes cluster without scripts so as to harness your skills.