Sometimes Kubernetes pods experience issues when trying to pull container images from a container registry. If an error occurs, the pod goes into the ImagePullBackOff state.

The status ImagePullBackOff means that a Pod couldn't start, because Kubernetes couldn't pull a container image. The 'BackOff' part means that Kubernetes will keep trying to pull the image, with an increasing delay (‘back-off’).

Here at Fixwebnode, we shall look into ways to resolve the ImagePullBackOff Kubernetes error.

More about the ImagePullBackOff Kubernetes error ?

The ImagePullBackOff error occurs when the image path is incorrect, the network fails, or the kubelet does not succeed in authenticating with the container registry. Kubernetes initially throws the ErrImagePull error, and then after retrying a few times, "pulls back" and schedules another download attempt. For each unsuccessful attempt, the delay increases exponentially, up to a maximum of 5 minutes.

To identify the ImagePullBackOff error: run the kubectl get pods command. 

The pod status will show the error like so:

NAME                    READY    STATUS              RESTARTS           AGE
fixwebnode-pod-1    0/1      ImagePullBackOff        0                     2m34s

Reasons for ImagePullBackOff Kubernetes error includes:

• The tag or image does not exist.
• There is a typo/spelling error in the tag or image name.
• The image registry needs authentication.
• The download limit on the registry has exceeded.

What to do when you experience ImagePullBackOff Kubernetes error ?

• When Pod specification provides the wrong repository name : Simply Edit pod specification and provide the correct registry.
• When Pod specification provides the wrong image name, or an unqualified image name: Simply Edit pod specification and provide the correct image name.
• When Pod specification provides an invalid tag, or fails to provide a tag: Simply Edit pod specification and provide the correct tag. If the image does not have a latest tag, you must provide a valid tag.
• When Container registry is not accessible: Simply Restore network connection and allow the pod to retry pulling the image.
• When The pod does not have the appropriate credentials to access the image: Simply Add a Secret with the appropriate credentials and reference it in the pod specification.

Steps to diagnose and resolve ImagePullBackOff and ErrImagePull Kubernetes Errors

1. Gather information

Run kubectl describe pod [name] and save the content to a text file for future reference:

$ kubectl describe pod [name] /tmp/troubleshooting_describe_pod.txt

2. Examine Events section in describe output

Check the Events section of the describe pod text file, and look for one of the following messages:

• Repository ... does not exist or no pull access
• Manifest ... not found
• authorization failed

3. Troubleshoot

i. If the error is Repository ... does not exist or no pull access:

This means that the repository specified in the pod does not exist in the Docker registry the cluster is using.

By default, images are pulled from Docker Hub, but your cluster may be using one or more private registries

The error may occur because the pod does not specify the correct repository name, or does not specify the correct fully qualified image name (e.g. username/imagename)

To resolve it, double check the pod specification and ensure that the repository and image are specified correctly.

If this still doesn't work, there may be a network issue preventing access to the container registry. Look in the describe pod text file to obtain the hostname of the Kubernetes node. Log into the node and try to download the image manually.

ii. If the error is Manifest ... not found:

This means that the specific version of the requested image was not found.

If you specified a tag, this means the tag was incorrect. Double-check that the tag in the pod specification is correct, and exists in the repository. Keep in mind that tags in the repo may have changed.

If you did not specify a tag, check if the image has a latest tag. Images that do not have a latest tag will not be returned, if you do not specify a valid tag. In this case, you must specify a valid tag.

iii. If the error is authorization failed:

The issue is that the container registry, or the specific image you requested, cannot be accessed using the credentials you provided.

To resolve this, create a Secret with the appropriate credentials, and reference the Secret in the pod specification.

If you already have a Secret with credentials, ensure those credentials have permission to access the required image, or grant access in the container repository.

[Need issues with fixing other Kubernetes errors ? We can help you. ]