Troubleshooting in Kubernetes
By DevOps Shack
25 Examples With Commands
1. Error: Unable to connect to the cluster
o Troubleshooting:
▪ Check kubeconfig file for correct cluster information.
▪ Verify network connectivity to the cluster.
o Example Commands:
kubectl config view
kubectl cluster-info
2. Error: Pod stuck in Pending state
o Troubleshooting:
▪ Check events for the pod using kubectl describe pod.
▪ Inspect the pod's YAML for resource constraints or affinity
issues.
o Example Commands:
kubectl describe pod <pod-name>
kubectl get events --namespace <namespace>
3. Error: Insufficient resources to schedule pod
Troubleshooting:
▪ Check resource requests and limits in the pod specification.
▪ Verify node resources using kubectl describe node.
o Example Commands:
kubectl describe pod <pod-name>
kubectl describe node <node-name>
4. Error: ImagePullBackOff
o Troubleshooting:
▪ Verify the image name and availability.
▪ Check image pull credentials using kubectl describe pod.
o Example Commands:
kubectl describe pod <pod-name>
kubectl get pods --namespace <namespace> -
o=jsonpath='{.items[*].status.containerStatuses[*].state}'
5. Error: CrashLoopBackOff
o Troubleshooting:
▪ Check container logs for details on the crash.
▪ Inspect pod events using kubectl describe pod.
o Example Commands:
kubectl logs <pod-name> <container-name>
kubectl describe pod <pod-name>
6. Error: Unauthorized access
o Troubleshooting:
▪ Verify RBAC permissions for the user.
▪ Check kubeconfig for correct credentials.
o Example Commands:
kubectl auth can-i --list
kubectl config view
7. Error: ConfigMap not updating in the pod
o Troubleshooting:
▪ Check if the ConfigMap is updated.
▪ Verify that the pod is configured to use the latest version.
Example Commands:
kubectl get configmap <configmap-name> -o yaml
kubectl describe pod <pod-name>
8. Error: Service not reachable
o Troubleshooting:
▪ Check service endpoints using kubectl describe service.
▪ Verify network policies and firewall rules.
o Example Commands:
kubectl describe service <service-name>
kubectl get networkpolicies
9. Error: Node not ready
o Troubleshooting:
▪ Check node status with kubectl get nodes.
▪ Review kubelet logs on the node for issues.
o Example Commands:
kubectl get nodes
kubectl describe node <node-name>
10. Error: PersistentVolumeClaim (PVC) pending
o Troubleshooting:
▪ Verify available storage in the cluster.
▪ Check storage class and provisioner.
o Example Commands:
kubectl get pvc
kubectl describe storageclass
11. Error: VolumeMounts not working in pod
o Troubleshooting:
▪ Check pod's YAML for correct volume mounts.
▪ Verify if the volume exists and is accessible.
o Example Commands:
kubectl describe pod <pod-name>
kubectl get pv
12. Error: Pod Security Policies (PSP) blocking pod
o Troubleshooting:
▪ Check PSP rules and RBAC for the pod.
▪ Inspect pod events using kubectl describe pod.
o Example Commands:
kubectl get psp
kubectl describe pod <pod-name>
13. Error: ServiceAccount permissions
o Troubleshooting:
▪ Verify ServiceAccount permissions using kubectl auth can-i.
▪ Check RBAC roles and role bindings.
o Example Commands:
kubectl auth can-i --list --
as=system:serviceaccount:<namespace>:<serviceaccount-name>
kubectl get roles,rolebindings --namespace <namespace>
14. Error: NodeSelector not working
o Troubleshooting:
▪ Check pod's YAML for correct node selector.
▪ Verify that nodes have the required labels.
o Example Commands:
kubectl describe pod <pod-name>
kubectl get nodes --show-labels
15. Error: Ingress not routing traffic
o Troubleshooting:
▪ Check Ingress resource for correct backend services.
▪ Verify that the Ingress controller is running.
o Example Commands:
kubectl describe ingress <ingress-name>
kubectl get pods --namespace <ingress-controller-namespace>
16. Error: Unable to scale deployment
o Troubleshooting:
▪ Verify available resources in the cluster.
▪ Check replica count in the deployment specification.
o Example Commands:
kubectl get deployments
kubectl describe deployment <deployment-name>
17. Error: Custom Resource Definition (CRD) not creating resources
o Troubleshooting:
▪ Check CRD definition for correct syntax.
▪ Verify controller logs for errors.
o Example Commands:
kubectl get crd
kubectl describe crd <crd-name>
18. Error: Pod in Terminating state
o Troubleshooting:
▪ Check for stuck finalizers in pod metadata.
▪ Force delete pod using kubectl delete pod --grace-period=0.
o Example Commands:
kubectl get pods --all-namespaces --field-
selector=status.phase=Terminating
kubectl delete pod <pod-name> --grace-period=0 –force
19. Error: Resource quota exceeded
o Troubleshooting:
▪ Check resource quotas for the namespace.
▪ Verify resource usage in the namespace.
o Example Commands:
kubectl describe quota --namespace <namespace>
kubectl top pods --namespace <namespace>
20. Error: Rolling update stuck or not progressing
o Troubleshooting:
▪ Check rollout status using kubectl rollout status.
▪ Verify image versions in the deployment.
o Example Commands:
kubectl rollout status deployment <deployment-name>
kubectl set image deployment/<deployment-name> <container-
name>=<new-image>
21. Error: Node draining or cordoning
o Troubleshooting:
▪ Check node conditions and events.
▪ Use kubectl drain with caution.
o Example Commands:
kubectl get nodes
kubectl describe node <node-name>
kubectl drain <node-name> --ignore-daemonsets
22. Error: Resource creation timeout
o Troubleshooting:
▪ Check for issues with the API server.
▪ Verify network connectivity to the API server.
o Example Commands:
kubectl get events --sort-by='.metadata.creationTimestamp'
kubectl describe pod <pod-name>
23. Error: Pod stuck in ContainerCreating state
o Troubleshooting:
▪ Check container runtime logs on the node.
▪ Inspect kubelet logs for errors.
o Example Commands:
kubectl get pods
kubectl describe pod <pod-name>
24. Error: Invalid YAML syntax
o Troubleshooting:
▪ Validate YAML syntax using online tools or linters.
▪ Check for indentation and formatting issues.
o Example Commands:
kubectl apply -f <file.yaml> --dry-run=client
25. Error: etcd cluster issues
o Troubleshooting:
▪ Check etcd logs for errors.
▪ Verify etcd cluster health.
o Example Commands:
kubectl get events --all-namespaces --field-
selector=involvedObject.kind=Pod,involvedObject.name=etcd
kubectl exec -it etcd-pod-name --namespace kube-system -- sh
etcdctl member list
etcdctl cluster-health
Remember to replace placeholders like <pod-name>, <namespace>, <deployment-name>,
etc., with actual values specific to your environment. Additionally, exercise caution
when using force deletion or draining nodes to avoid potential data loss or service
disruption.
