2. Developing at GitHub

2.1. Overview

The VisIt project has a number of repositories located at the GitHub visit-dav organization.

The primary repository for doing VisIt development is the visit repository.

The following top level directories exist in the visit repository.

data - Data files used by the test suite.
docs - Legacy documentation including design documents and presentations.
scripts - Various scripts used for doing VisIt developement including scripts for managing docker containers and doing continuous integration.
src - The VisIt source code. It includes the Read the Docs documentation and the regression test suite.
test/baseline - The baseline results for the regression test suite.

2.2. Setting Up Git LFS

Git LFS (Large File Storage) is a mechanism to help revision control large files efficiently with git. Instead of storing large files in the repo, LFS provides an extension that stores small text files with metadata in the repo and the actual files on another server. These metadata files are called “pointer” files. We use LFS for binary data including our test data tar files, source code for third party libraries, and regression test baseline images.

Git LFS is not part of the standard git client. See https://git-lfs.github.com/ for how to obtain Git LFS.

When installing, use the following option:

git lfs install --force --skip-smudge

The “skip smudge” command sets up LFS in a way that skips automatically pulling our large files on clone. We do this to conserve bandwidth.

To obtain these files you will need to do some extra incantations followed by an explicit:

git lfs pull

For more details about using Git LFS, read our additional notes.

2.3. Accessing GitHub

The following link points to a page for creating a personal access token to use for the password when accessing GitHub through the command line. Use the following scopes for the token:

repo:status

repo_deployment

public_repo

https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/

The following link describes how to add your ssh key to your GitHub account.

https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/

2.4. Cloning the Repository and Setting Up Hooks

You can access GitHub either through https or ssh. If you use https you will be prompted for your password whenever you push to GitHub. There are ways you can have your password cached for a period of time to reduce the frequency of entering your password. However, if you have two-factor authentication set up you will need to create a personal access token to use in place of the password. If you use the ssh protocol you can set things up so that you never have to enter a password by adding your ssh key to your GitHub account.

To clone the repository:

git clone --recursive https://github.com/visit-dav/visit.git

or:

git clone  --recursive ssh://git@github.com/visit-dav/visit.git

If for some reason the --recursive flag was overlooked when the repository was originally cloned, this can be easily remedied by:

cd visit
git submodule init
git submodule update

To setup our hooks:

cd visit
./scripts/git-hooks/install-hooks.sh

2.5. Sparse Checkouts

If your workflow includes multiple clones of the repository in order to work on different VisIt tasks without having to constantly swap branches in a single clone, sparse checkouts might be a good fit. Sparse checkouts allow you to clone the repo, but only download specific directories of interest. For instance, if you don’t have a need to regularly run the regression tests, you might only want the src and scripts directories.

Here’s how to clone and checkout only those directories:

git clone --sparse --recursive https://github.com/visit-dav/visit.git
git sparse-checkout set 'src/*' 'scripts/*'
git checkout

The use of the asterisk in the set command above isn’t technically necessary, but does guard against other subdirectories with the same name from being part of the checkout. For instance, to get only the data directory where the test data lives, if git sparse-checkout set 'data' is used, then both the data and the src directories will be downloaded because there is a data subdirectory in src/tools/data.

You can use the same commands if you prefer ssh over https for cloning. Branching and development in sparse checkouts works the same as with the full repository.

If later you want to run regressions, you can create a new full clone or a separte sparse clone of just the data or test directory, or both. If the test data directory is not located peer to src, you can tell VisIt’s CMake build system where to find it by setting VISIT_DATA_DIR to the location of data before you configure with CMake. The location of the test baselines can be passed to the regression test script via -b /path/to/test/baseline on the command line when running tests.

2.6. Creating a Branch

Development for VisIt is done off of two main branches, the develop branch and the current release candidate branch, which was 3.2RC when this content was written. The develop branch is used for development that will go into the next major or minor release. Major releases are releases where the first digit of the release number is incremented, Minor releases are releases where the second digit of the release number is incremented. The release candidate branch is used for development that will go into the next patch release. Patch releases are releases where the third digit of the release number is incremented.

There is no convention on the names of a branch. One commonly used convention is task\Username\YYYY_MM_DD_Description where Username is your GitHub user name, YYYY is the current year, MM is the current month, DD is the current day, and Description is a short description of the task to be performed. Since branches only exist while you are doing the development, the name isn’t critical, but it should be sufficiently descriptive so that someone can have some idea what the development on the branch is about.

To create a branch off of the develop branch:

git checkout develop
git pull
git checkout -b task/user/2021_05_07_bug_fix

To create a branch off of the current release candidate:

git checkout 3.2RC
git pull
git checkout -b task/user/2021_05_07_bug_fix

When you switch branches, you may also need to update submodules so they match your branch:

git submodule update

2.7. Doing Development

Doing development using the Git version control system can be complex and take considerable time and effort to master. The primer below is just meant to get you started in modifying files and then pushing the changes to GitHub so that they can be integrated into VisIt.

To add a new file or modify an existing file, edit the file with your favorite text editor and then use the add command so that git knows you want the file to be part of your next commit. To add a file:

git add src/myfile

To delete an existing file use the rm command:

git rm src/myfile

Once you have modified one or more files you can commit the change to git. You will typically do a commit after having modified one or more files that completes a logical unit of change. To commit the added files with a comment:

git commit -m "Description of my change."

It is recommended that you make commits frequently so that you can better track individual changes. The commit descriptions are typically brief. The record of the individual commits will not go into the final record of the commit, since we do “Squash and Merge” commits that merge all the commits into a single commit at GitHub. The individual commits will be helpful to you as a developer if you need to go back and understand when making many changes over a period of time. It may also potentially make it easier for reviewers to understand your commits.

Once you have finished all your changes you can push the change to GitHub. To push your changes to GitHub:

git push --set-upstream origin task/user/2021_05_07_bug_fix

Once you have pushed your changes to GitHub, you can submit a pull request.

2.8. CMake Build System

VisIt’s build system uses BLT CMake helpers. BLT is included in VisIt’s git repo as a git submodule. To obtain the submodule, use git clone –recursive when cloning, or manually setup the submodule after cloning using:

git submodule init
git submodule update

When you switch branches, you may also need to update submodules so they match your branch:

git submodule update

Branch development with git submodules can lead to unintended submodule commits. To avoid this, we have an CI check that ensures the active submodule commits match a version explicitly listed in a hashes.txt file at the root of the git repo.

2.9. GitHub Administration

The VisIt project uses some additional, paid, services on GitHub. For example, we use additional data packs, for higher storage and bandwidth thresholds than the free tier. These are handled under the VisIt organization’s Billing and Plans section of the organization Settings. These are currently billed annually and renew in mid-January (January 19th). In the past, someone in LLNL’s SD IT department would feed GitHub a credit card number to fund the services for another year. More recently, LLNL requires a more formal paper trail for anything billed in this manner. So, to ensure someone in LLNL’S SD IT department actually takes care of it, a ServiceNow ticket needs to get generated and it needs to be one that specifically goes to the SD IT procurement team. As of this writing, SD IT Procurement can be found in Service Now by going to Home->Business Applications Catalog-> Strategic Deterrence (SD)->SD Procurement. Be sure to set Urgency to 1-High, provide VisIt’s project/task number and then indicate in the description it is… “For the yearly renewal of ‘5 Git LFS data packs’, additional storage and bandwidth needed by our project on GitHub (github.com). Tiffany Taylor (taylor253) has been the SD IT department member to take care of it.