I had a research project where I wanted to use synethic data to help address the AI knowledge cutoff problem. To do that the key component was using Ollama to distill questions from projects source code to a smaller model. The problem you’ll run into is that all the large models that can produce higher quality data require more than memory than a 5090ti or 7900xtx comes with. While you could just buy AWS credits or an out of stock Nvidia Pro. Including I didn’t have funding for this project, so API access to Claude wouldn’t be possible either.

Luckly if you’re in univeristy, most have free resources for students and researchers through their CS department. However since it’s shared its not like you can just login and run your project that takes 3 days to run. You have to wait in line like everyone else or buy private hardware. Most systems use SLURM which is what we will be using as well for this post.

SLURM

In case you’re new to SLURM, here’s a quick abridge guide to slurm. We’re be using the srun and sbatch commands. srun is a command that submits the foreground command to the queue. Often I find myself using this command to download datasets since they only need 1 CPU and core, and those jobs finish fast. sbatch is the command to submit a script to the queue, these are always scripts. For example you could submit Python, R or MatLab scripts to a queue, but in real life the vast majority are short bash scripts. Typically my job’s settings, time, paritions, amount of CPUs, amount of CPU cores and maybe amount of GPUs, then a one line executable that runs.

Modules

Better run clusters will have huge arrays of modules. For Ollama we have these versions to choose from.

$ module spider ollama

  ollama:
     Versions:
        ollama/0.5.13
        ollama/0.11.3
        ollama/0.12.5
        ollama/0.13.1

  For detailed information about a specific "ollama" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider ollama/0.13.1

Great! This is exactly what we need right? Version 0.13.1 is only 6 months old at the time of writting this. Wait Gemma 4 and any other newer model requires Ollama v0.20.0-rc0 or later… The landscape is just moving that fast, so we can’t rely on vendored solutions even if they exist. So lets see if containers can help us solve this solution.

Docker

Most people are most familiar with docker, however in this environment that would be the naive apporach. Well Ollama Dockerhub gives us a command we can run start it. Let’s run it.

$ docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
-bash: docker: command not found

Hm That didn’t work, this is the error I got on my univeristy’s cluster.

What if we sent it to the slurm queue? Maybe its installed on the nodes but not the head node?

$ srun docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
srun: job 125198 queued and waiting for resources
srun: job 125198 has been allocated resources
slurmstepd: error: execve(): docker: No such file or directory
srun: error: node19: task 0: Exited with exit code 2

You have now experienced the first lesson most developers new to clusters learn. Docker is not installed, and will never be installed. From my experience, don’t even ask. The reason for this is because docker has a system daemon that runs as root. This is to manage the containers like a backend of a web server. The concern is that if there is a CVE exploit for docker, then every cluster will be exposed at once, a list of known exploits is listed here. This problem also leaks into the docker group where you could bind / for example and chroot into it.

Podman

Made by Red Hat, Podman is a drop-in replacement for Docker. The reason you would use podman is for rootless containers. Often in more security focused environments, but they are typically just headaches to deal with. However most clusters do have podman installed so this is a option.

Lets see!

$ srun podman run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
srun: job 125199 queued and waiting for resources
srun: job 125199 has been allocated resources
time="2026-04-26T20:50:52-05:00" level=error msg="stat /run/user/17916: no such file or directory"
srun: error: node19: task 0: Exited with exit code 1

Almost! But look we got a log from ollama. We can tell because its formatted differently. time="2026-04-26T20:50:52-05:00" level=error msg="stat /run/user/17916: no such file or directory"

Lets try again with some rootless container magic, this is just binding id’s to the correct places. Basically Podman is not designed to be used on clusters. It can be used, and it will work but it is not the correct tool for the task. Unless you require Docker’s layer system for example. It could make sense if you need to compile LibTorch for example or another massive library.

Another thing to remember is that your experience with Podman will depend on the quality of your Sysadmin.

$ srun bash -c 'export XDG_RUNTIME_DIR=/tmp/run-$UID && mkdir -p $XDG_RUNTIME_DIR && chmod 700 $XDG_RUNTIME_DIR && podman run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama'
srun: job 125201 queued and waiting for resources
srun: job 125201 has been allocated resources
time="2026-04-26T21:19:23-05:00" level=error msg="cannot find UID/GID for user w180cxr: No subuid ranges found for user \"w180cxr\" in /etc/subuid - check rootless mode in man pages."
time="2026-04-26T21:19:23-05:00" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
time="2026-04-26T21:19:24-05:00" level=warning msg="Network file system detected as backing store.  Enforcing overlay option `force_mask=\"700\"`.  Add it to storage.conf to silence this warning"
Resolving "ollama/ollama" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull registry.fedoraproject.org/ollama/ollama:latest...
Trying to pull registry.access.redhat.com/ollama/ollama:latest...
Trying to pull registry.centos.org/ollama/ollama:latest...
Trying to pull docker.io/ollama/ollama:latest...
Getting image source signatures
Copying blob sha256:d7dd4ba38b82a932fed123582fbb21c7b459edb6b93102413fe091a8d1dd3c11
Copying blob sha256:2d0e9b7d523c69fc07c4712a7336eea5f23aa5f6660a47d825cbdff978096e38
Copying blob sha256:e97c594ed5ac959861c3c6e0158e9c72df513aa86429d1a6e27443c0ef198e57
Copying blob sha256:b40150c1c2717d324cdb17278c8efdfa4dfcd2ffe083e976f0bcedf31115f081
Copying blob sha256:d7dd4ba38b82a932fed123582fbb21c7b459edb6b93102413fe091a8d1dd3c11
Copying blob sha256:2d0e9b7d523c69fc07c4712a7336eea5f23aa5f6660a47d825cbdff978096e38
Copying blob sha256:e97c594ed5ac959861c3c6e0158e9c72df513aa86429d1a6e27443c0ef198e57
Copying blob sha256:b40150c1c2717d324cdb17278c8efdfa4dfcd2ffe083e976f0bcedf31115f081

Error: 4 errors occurred while pulling:
 * initializing source docker://registry.fedoraproject.org/ollama/ollama:latest: reading manifest latest in registry.fedoraproject.org/ollama/ollama: manifest unknown: manifest unknown
 * initializing source docker://registry.access.redhat.com/ollama/ollama:latest: reading manifest latest in registry.access.redhat.com/ollama/ollama: name unknown: Repo not found
 * initializing source docker://registry.centos.org/ollama/ollama:latest: pinging container registry registry.centos.org: Get "https://registry.centos.org/v2/": dial tcp: lookup registry.centos.org on 10.1.1.254:53: no such host
 * writing blob: adding layer with blob "sha256:b40150c1c2717d324cdb17278c8efdfa4dfcd2ffe083e976f0bcedf31115f081": Error processing tar file(exit status 1): lsetxattr /: operation not supported
srun: error: node19: task 0: Exited with exit code 125

Well we tried. This is why I don’t recommend Podman either. I’m sure it is possible, but not a good use of time.

Apptainer

Apptainer is if podman actually worked and was almost as easy to use as Docker. Formerly Singualrity, it is the same tool (for us) if you see it instead. It is completely different from both Docker and Podman, there are no layers or composes.

Defination

You can also create sandboxes as well, however we’ve be using their defination files. This is just a Dockerfile but for Apptainer, however the synatx are completely different so lets review them.

Bootstrap: docker
From: ollama/ollama:{{ OLLAMA_VERSION }}

%arguments
    OLLAMA_VERSION=0.21.2

%post
    mkdir -p /root/.ollama

%environment
    export OLLAMA_HOST=0.0.0.0:11434
    export OLLAMA_MODELS=/root/.ollama

%runscript
    echo "Starting Ollama server, the default host is $OLLAMA_HOST"
    exec ollama serve

%labels
    Author redhawk
    Reference https://redhawkcodes.dev

%help
    This container runs an Ollama server.
    To use it on HPC with GPU support, run:
    apptainer run --nv -B /path/to/models:/root/.ollama ollama.sif

Basically we are literally repacking the Dockerhub Ollama container. It’s that simple! Lets build it with this command

apptainer build ollama.sif ollama.def

For reference its apptainer build output.sif input.def since I’ve also thought that is confusing.

Security

Some clusters have Apptainer disabled for building containers, so either prepend srun or transfer it to the cluster. I’ve had trouble with this in the past, so watch out!

Running

Okay great! Our container is built and it should be done soon. So how do we run this on the A100 or the H100 my univeristy keeps talking about? We’ll use a bash script and the sbatch command to submit it to the queue. Remember sharing is caring 😍.

#!/bin/bash
#SBATCH --job-name=ollama-gemma4
#SBATCH --partition=a100
#SBATCH --gres=gpu:1
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --time=1-00:00:00
#SBATCH --output=out.log

set -e

# echo "Starting autollm on $(hostname)" | notify_slack
export NO_COLOR=1
nvidia-smi

# echo "Starting ollama example on $(hostname)" | notify_slack
apptainer run --fakeroot --nv -B "$HOME/.ollama/models:/root/.ollama" hpc/ollama.sif &
sleep 15

# 1. Pull the model
# In real life you shouldn't stall a GPU node with downloading anything,
# but for the simplicy of this post I didn't do that here.
curl http://localhost:11434/api/pull -d '{
  "model": "gemma4:31b"
}'

# 2. Ask it who it is
curl http://localhost:11434/api/chat -d '{
  "model": "gemma4:31b",
  "stream": false,
  "messages": [
    {"role": "user", "content": "What is your name and who made you?"}
  ]
}'

# echo "Finished ollama example on $(hostname)" | notify_slack

Lets break this down into sections.

SLURM job settings

We have a job that uses bash as the interpreter, named ollama-gemma4. It will run on the a100 partition or a superior one if available. We are requesting a single GPU and 16 CPU cores PER node. Also requesting a 1 node with a timelimit of 1 day and our output file is named out.log. Tasks are for libraries like MPI, don’t worry about that.

Some things to note, there is no way to request GPU memory requirement like you can with system Ram requirements. Often times you’ll see paritions like a100-40 and a100-80 respectifully for the 40 and 80 gigabyte versions of the card.

Script

First we are exiting on any errors. Second I commented it out, but I also use notify_slack which might also be helpful to you. It just sends a webhook to slack to notify me in real time. Then we get our GPU infomation, sometimes we get more than what we requested. For example you could get an 80 gigabyte a100, and the next run you get a 40 gigabyte a100, even if you only required the 40 gigabyte varient. Third we start the Ollama’s apptainer, on older systems try removing --fakeroot since that can cause problems. Then make sure you create the directory with this command.

mkdir $HOME/.ollama/models

After that we can access Ollama via their API. For the example we are using curl for simplicy. However you can replace that with any program that interacts with Ollama’s API. The container is started on the default host of localhost:11434.

Submitting

After you create that file, first we need to make it a script. So lets run

chmod +x ollama.sbatch

After we made it a script we can submit it.

sbatch ollama.sbatch

Now you’re done, just wait until its your turn. Obvious make sure your container’s file is in the same directory and check if there are any errors.

Results

I get this response back.

{
   "model":"gemma4:31b",
   "created_at":"2026-04-27T03:32:11.313781563Z",
   "message":{
      "role":"assistant",
      "content":"I do not have a name. I am a large language model, trained by Google.",
      "thinking":"*   Question 1: \"What is your name?\"\n    *   Question 2: \"Who made you?\"\n\n    *   I am a large language model trained by Google.\n\n    *   *Name:* I don't have a human name. I am a model developed by Google. (Wait, some might call me Gemini, but usually, the standard identification is \"a large language model trained by Google\").\n    *   *Creator:* Google."

   },
   "done":true,
   "done_reason":"stop",
   "total_duration":30931353706,
   "load_duration":27932225428,
   "prompt_eval_count":24,
   "prompt_eval_duration":61418802,
   "eval_count":120,
   "eval_duration":2764190436
}

Hello “a large language model”, I think thats a odd name but thats a success for container and sbatch file!

Conclusion

I tried to keep this brief and meat-packed, but there’s a lot to explain and no one to ask for help. Whenever a landscape moves to fast you have to figure out how to vendor your own software like this for example. Hope this helps! Please cite this post if you’re working on a publication and good luck!