CoffeeBeforeArch.github.io

View on GitHub

Working with GPGPU-Sim - Introduction

GPGPU-Sim is one of the most highly cited GPGPU microarchitecture simulators. This series of tutorials will introduce you to the basics of working with GPGPU-Sim. Feel free to message me if you have any questions.

While the original GPGPU-Sim repository is still being updated, a large portion of the changes come from the fork at Purdue University. This is maintained/developed by the AALP lab of Dr. Timothy Rogers. A link to this fork is provided below, and will be the repository used for these tutorials.

My Environment Setup

Below are some details about my local development environment. GPGPU-Sim compiles with a number of different GCC/G++ and CUDA SDK versions. One thing to note is that not all GCC/G++ versions work with all CUDA SDK versions (there is typically both a minimum and maximum release supported). While I typically use the latest GCC/G++ (10 at the time of writing this) for my work, the maximum supported GCC/G++ version for CUDA 9.1 is 6.

Software Version
Operating System Ubuntu 18.04
GCC/G++ Version 6.5
CUDA SDK Version 9.1

CUDA SDK Installation

I downloaded the CUDA 9.1 SDK from the here, and followed the installation instructions. While navigating the command line installer, I did not install the driver. The driver is not required for you to use GPGPU-Sim.

Compiling GPGPU-Sim

Before You Continue…

Before you do anything, read the GPGPU-Sim README. It will tell you the dependencies for building the simulator. Most build errors people run into come from not installing the listed dependencies.

Secondly, we will be using the dev branch for the simulator. This is where all of the development of the GPGPU-Sim is going on. The master branch has not been updated for many years. You can switch to the dev branch of the simulator using git checkout dev from within the GPGPU-Sim directory.

Basic Steps

To get started building GPGPU-Sim, you must first run the setup_environment script. This can be done with:

source setup_environment

However, you may be met with this error message:

GPGPU-Sim version 4.0.0 (build gpgpu-sim_git-commit-0748a2aa0325ec2c9f8f011967103525b0237c87-modified_0.0) ERROR ** Install CUDA Toolkit and set CUDA_INSTALL_PATH.

This just means that the envornment setup script can’t find the CUDA SDK. You just need to set the CUDA_INSTALL_PATH environment variable to the base of where the CUDA SDK was installed. Here is what I did on my system:

export CUDA_INSTALL_PATH=/usr/local/cuda/

Now you should be met a bunch of text that ends with the main thing we’re looking for: setup_environment succeeded. From here, we can build GPGPU-Sim using make. However, you will likely want to parallelize the build using something like make -j4 (the number 4 can be replaced with a different number depending on how you want to parallelize the build).

Common Errors

Makedepend Error

If you run into an error at src/cuda-sim/Makefile:150 that says that the Makefile.makedepend file can not be found, you may need to install xutils-dev. This can be done with the following line:

sudo apt-get install xutils-dev

Bison Double Free Error

If you run into an error like this:

bison --name-prefix=ptx_ -v -d ptx.y --file-prefix=/home/cba/forked_repos/gpgpu_arch_tutorial/gpgpu-sim_distribution/build/gcc-/cuda-10020/release/cuda-sim/ptx
ptx.y:36 parser name defined to default :"parse"
"ptx.y", line 37: junk after `%%' in definition section
"ptx.y", line 37: no input grammar
"ptx.y", line 37: unknown character with code 0x0 in declaration section
double free or corruption (!prev)

This likely indicates that you did not install flex and/or bison. That can be done with the following line:

sudo apt-get install flex bison

Testing Your Build

To test the build, I’ll be using a simple version of matrix multiplication with a small input (remember, simulation takes orders of magnitude longer than running on silicon, so relatively small inputs can still take hours to complete). The code comes from my teaching series on CUDA programming, and can be found here. I simply changed the square matrix dimension, specified at the start of the main function, from 2^10 to 2^5.

Compiling Matrix Multiplication

We can compile our matrix multiplication using NVCC. NVCC comes with the CUDA SDK we installed earlier. I compiled my code like this:

nvcc mmul.cu -lcudart -o mmul

Linking against libcudart.so is a necessary part to using GPGPU-Sim (GPGPU-Sim fundamentally just intercepts calls to the NVIDIA CUDA runtime library). I’ll discuss more about how this works in a later post. You can check what shared objects an application uses with ldd.

ldd mmul

You should see that libcudart.so.9.1 (on my machine) is being loaded from the GPGPU-Sim build, and not from the CUDA SDK directory. Here is my output:

libcudart.so.9.1 => /home/cba/forked_repos/gpgpu_arch_tutorial/gpgpu-sim_distribution/lib/gcc-6.5.0/cuda-9010/release/libcudart.so.9.1 (0x00007faab5593000)

Running Matrix Multiplication with GPGPU-Sim

The last thing we need to do to run our simulation is to bring the GPGPU-Sim configuration files into the same directory as the application. These configure GPGPU-Sim to model different architectures. For this test, I’ll use the configuration files for an NVIDIA TITAN V GPU. This can be found at $GPGPUSIM_ROOT/configs/tested-cfgs/SM7_TITANV/.

Copy both config_volta_islip.icnt and gpgpusim.config into the same directory as your matrix multiplication executable. The only thing left to do is to run the application. When you run the application, the simulator starts automatically. When the application completes, you will see a number of statistics printed out to the screen using stdout. If you want to parse through these stats, I would suggest redirecting the output to a text file.

./mmul > out.txt

Concluding Remarks

Congratulations! You’ve taken your first step towards doing GPU architecture research! In later posts, I’ll discuss notable parts of the codebase, show how to make meaningful modifications, and show off notable features. Feel free to message me with any questions

Cheers,

–Nick