17. Docker Containers For CI Testing
We use Azure Pipelines for CI testing VisIt’s Pull Requests, located at VisIt Azure DevOps Space.
To speed up our CI testing we use Docker containers with pre-built third party
libraries (TPLs). These containers leverage our build_visit
third party
build process. The Docker files and build scripts used to create
these containers are in scripts/ci/docker
. The process to create the
container varies somewhat if you have Docker installed on the same
system as the git checkout of your branch.
17.1. Creating the container with Docker on the same system as the git checkout
Create the container using build_docker_visit_ci.py
.
cd scripts/ci/docker
python build_docker_visit_ci.py
This creates the container with a tag that will include today’s date and a short substring of the current git hash.
Example Tag: visitdav/visit-ci-develop:2020-11-11-sha433ef0
This will typically take several hours to complete.
17.2. Creating the container without Docker on the same system as the git checkout
Create two tar files and the Docker command to create the container using
build_docker_visit_ci.py
.
cd scripts/ci/docker
python build_docker_visit_ci.py
This will create the files
visit.build_visit.docker.src.tar
visit.masonry.docker.src.tar
The command will also fail with output similar to this
[exe: git rev-parse HEAD]
[exe: docker build -t visitdav/visit-ci-develop:2020-11-11-sha433ef0 . --squash]
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
You will use the docker build ...
command to build the container on
the system you have Docker installed.
You can now move over to the system where you have Docker installed. Bring up up a shell window and create a new directory or folder to contain the files necessary to create the container.
mkdir docker_ci
cd docker_ci
Now copy all the files in scripts/ci/docker
to your new folder. Now
you can run the docker build ...
command to create the container. For
example:
docker build -t visitdav/visit-ci-develop:2020-11-11-sha433ef0 . --squash
This will typically take several hours to complete.
17.3. Push the container to Dockerhub
Now that you have created your Docker container image, you are ready to push
it to VisIt’s DockerHub Registry
using docker push <container-name>
.
If you do not already have a DockerHub account, go here and sign up for one. Then contact another member of visitdav and ask to be added to the organization.
You will need to be logged into DockerHub to successfully push. Here is an example push command:
docker login docker.io
docker push visitdav/visit-ci-develop:2020-11-11-sha433ef0
17.4. Update VisIt to use the new Docker image
To change which Docker Image is used by Azure, edit azure-pipelines.yml
and change the container_tag variable.
#####
# TO USE A NEW CONTAINER, UPDATE TAG NAME HERE AS PART OF YOUR PR!
#####
variables:
container_tag: visitdav/visit-ci-develop:2020-12-09-shaf6ef22
If you change the operating system, you will need to update the vmImage variable. It is specified in two locations.
pool:
vmImage: 'ubuntu-18.04'
When the PR is merged, the Azure changes will be merged and PRs to develop will now use the new container.