Podman

We'll use podman to deploy the various components of this stack, but in a production environment, an enterprise-grade scheduler would be more appropriate (ex. Kubernetes).

Overview

Podman is a container runtime that provides a lightweight and secure way to run containers. It is designed to be a drop-in replacement for Docker, but with a smaller footprint and better performance. It is widely used in Linux distributions.

Up and Running

Installation Options:

Podman Desktop: https://podman-desktop.io/downloads
Podman CLI: https://podman.io/docs/installation

Podman Desktop Installation Docs:

On Enterprise Linux-based distros (ex. RHEL, Fedora, Rocky, ...), do the following:

sudo dnf -y install podman
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
sudo loginctl enable-linger

On Mac and Windows, you may need to increase the memory of the podman machine to allow the stack to run properly.

podman machine stop
podman machine set --memory 8192
podman machine start

Setup podman network, directory for files, and pull the images used in this tutorial:

export LLM_STACK_DIR=$HOME/llm-stack
mkdir -p $LLM_STACK_DIR/
podman network create llm-stack
podman pull \
  pgvector/pgvector:pg17 \
  ollama/ollama:0.9.6 \
  ghcr.io/berriai/litellm:v1.74.0-stable \
  redis:8.0 \
  apache/tika:3.2.1.0 \
  ghcr.io/open-webui/open-webui:v0.6.18