Local Text GenAI

Abstract

If you’ve ever wanted to leverage your own “ChatGPT” at home, you have the ability with Ollama. I’ve included the instructions utilizing Docker and NVidia GPU(s).

Requirements

Hardware

Server

NVidia GPU (8gb VRAM Minimum)

Software

Docker

Debian/Ubuntu

Proxmox 8.x (Optional)

Portainer (Optional)

NVidia GPU (Optional)

Docker Files Used

• Backend Ollama
• Backend GitHub
• Frontend Open WebUI

Guides

• Open WebUI - Quickstart
• Find Models
• How to Run Llama 3 Locally with Ollama and Open WebUI

How to get GPU working

Install NVidia CUDA Toolkit

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-8

sudo apt-get install -y nvidia-open

sudo apt-key del 7fa2af80

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb

sudo dpkg -i cuda-keyring_1.1-1_all.deb

sudo apt-get update

sudo apt-get install cuda-toolkit

sudo apt-get install nvidia-gds -y

sudo reboot

Install GPU Drivers on Host

Installing NVidia Linux Drivers

sudo apt install gcc -y

Install NVidia Container Toolkit

NVidia Container Toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update

sudo apt-get install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

nvidia-ctk runtime configure --runtime=docker --config=$HOME/.config/docker/daemon.json

systemctl --user restart docker

sudo nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place

Validate it works

``` sudo docker run –rm –runtime=nvidia –gpus all ubuntu nvidia-smi