Optimize Docker Pull for your Gitlab Pipelines

A guide to optimize docker pull for your gitlab pipeline

Kunal Saini
Kunal Saini
July 06, 2021
3 min read

Send Transactional emails With Confidence

Our dedicated support team works around the clock because transactional emails never stop.

Free Sign UpSet up in 2 minutes . No Credit Card Required

Things shifted slightly in the Cloud Native world a while back, when the Docker Hub turned on rate limiting. If you run a Kubernetes cluster or Gitlab pipelines, or make extensive use of Docker images or Docker build, this is something you need to be aware of as it could cause outages.

Docker pull comes with a limit of pull which is 100 requests per 6 hours, so if your pipeline pulls more than that limit you will encounter the pull limit error from docker.

toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

error

Prerequisites:

  • SSH to Gitlab Runner Machine
  • Article assumes that you have an understanding of docker
  • Your GitLab runner executor is docker

To resolve the error we have two approaches:

  1. Use a free docker account that comes with a 200 request per 6 hours limit or a paid docker account. this is fairly simple and requires only docker login before pulling an image

    echo $DOCKER_PASSWORD | docker login -u $DOCKER_USER --password-stdin
    docker build .
  2. Set policy to your GitLab runner to pull if not present

    Pull policy will always first look for the already present image and will use it if present otherwise falls back to docker pull.

    [runners.docker]
      tls_verify = false
      image = "docker:dind"
      privileged = true
      disable_entrypoint_overwrite = false
      oom_kill_disable = false
      disable_cache = false
      volumes = ["/cache"]
      shm_size = 0
      tls_cert_path = ""
      pull_policy = "if-not-present"

This works great but what if you build a docker image inside your docker executor (docker inside docker), docker build will not be able to find out the images that are available on your runner machine because we are executing the build inside a docker container.

One way would be to solve this by using the docker private registry.

Setting up docker private registry:

  1. SSH to the runner machine
  2. Start a registry container

    Use the following to start the registry container:

    docker run -d -p 5000:5000 --restart=always --name registry registry:2
  3. Copy an image from Docker Hub to your registry

    You can pull an image from Docker Hub and push it to your local registry. The following example pulls the golang alpine image from Docker Hub and then pushes it to the local registry.

    docker pull golang:1.16-alpine

    Tag the image as localhost:5000/golang:1.16-alpine. This creates an additional tag for the existing image. When the first part of the tag is a hostname and port, Docker interprets this as a registry location when pushing.

    docker tag golang:1.16-alpine localhost:5000/golang:1.16-alpine

    Push the image to the local registry running at localhost:5000:

    docker push localhost:5000/golang:1.16-alpine

You should not expose your local registry to the internet for security reasons, we will be using the private IP of your runner's machine to access this private registry inside the Docker executor

  1. Update entry-point of .gitlab-ci.yml file

    Update your .gitlab-ci.yml file to allow the insecure local registry, following is an example:

    build:
      stage: build
      image: docker:20.10.5
      tags:
      services:
        - name: docker:20.10.5-dind
          command: ["--insecure-registry=<PRIVATE_IP_ADDRESS>:5000"]
      variables:
        DOCKER_HOST: tcp://docker:2375/
        DOCKER_DRIVER: overlay2
        DOCKER_TLS_CERTDIR: ""
  2. Pull image from the local registry before building docker image:

    Following in the .gitlab-ci.yml file would pull the golang alpine image from the local registry and then tag it.

    before_script:
      - docker pull <PRIVATE_IP_ADDRESS>:5000/golang:1.16-alpine
      - docker tag <PRIVATE_IP_ADDRESS>:5000/golang:1.16-alpine golang:1.16-alpine

    Now your docker build command should use this locally pulled image instead of pulling from Docker Hub. You may validate it by verifying the image version in your Dockerfile

    FROM golang:1.16-alpine AS builder

Limitation of the local registry approach:

Your CI file needs to be updated to pull the image from the local registry, which might cause issues if you have multiple GitLab runners running or configured.

I hope you find this helpful. You can provide feedback or suggestions in the comment section below. Cheers!!!



Mailazy Docs

Integrate with Transactional email service in minutes

click here

Most Popular Tags

EngineeringSMTPShort ReadAPIsBest PracticesEmail SecurityEmailEmail TemplatesGolangMarketing Email



What is Mailazy?

Mailazy is a Transactional Email Platform specially built for developers which satisfies the requirement for use cases like Reset Password Emails, OTP Emails, Welcome Emails, and so on. Mailazy has built to serve their customers a high value transactional email service. Sending transactional emails shouldn’t be hard with the right transaction email service. With an email service provider or simple transactional email service like Mailazy, you can easily send and track your transactional emails. Our goal is to help you achieve high delivery rates and keep your transactional emails out of the spam folder. We are always there for whenever ypu need us.

Lets get started!

Kunal Saini

Kunal Saini

Kunal is a Lead Software Development Engineer at Mailazy. He graduated in Computer Science and considers himself a learner of life. He loves to play cricket, football and computer games.

View Profile