WhichLLM Guide

IMPLEMENTATION GUIDE

Local LLM Setup Guide

whichllm, vLLM, Ollama, and Hermes Agent

A clean operating guide for installing the model recommender, choosing a local writer model, serving models with vLLM, running models with Ollama, and launching Hermes Agent with Hermes 3. Field Detail Prepared May 20, 2026 Primary source

Andyyyy64/whichllm README, checked at commit 282e77b64a9202475688c5d9941c824d02d84b4c

Target systems Windows 10 or later, macOS 14 or later for Ollama, Apple Silicon for vLLM-Metal, and modern Linux distributions

PDF standard

No browser header/footer artifacts, no running page furniture, fixed margins, aligned tables, readable code blocks

RECOMMENDED PATH

Use whichllm to select a model for your hardware. Use Ollama for the simplest local chat and Hermes Agent setup. Use vLLM when you need an OpenAI-compatible server, batching, or production-style serving.

1. How the Pieces Fit Together

These tools solve different parts of the local LLM workflow. Keep this split clear before installing anything. Tool Role Use it for Anchor command whichllm Model

recommender CLI

Detects hardware and ranks Hugging Face models that

should fit your CPU, GPU, RAM, and VRAM. whichllm –profile general –top 5 vLLM Model serving engine

Runs an OpenAI-compatible API server, mainly for Linux

GPU deployments and higher-throughput serving. vllm serve <model>

Ollama

Local model runner Installs models by simple names and exposes a local API on port 11434.

ollama run hermes3
Hermes 3 /
Hermes

Agent Agent model and app launch

Hermes 3 is the Nous Research model family. Hermes Agent

can be launched through Ollama with Hermes 3.

ollama launch hermes

–model hermes3 Working sequence 1 Install whichllm and run a hardware-aware recommendation. 2 Pick the workflow: Ollama for local chat and agents, or vLLM for an API server. 3 Install Ollama and pull the selected model name, or install vLLM and serve the Hugging Face model ID. 4 For Hermes Agent, install Ollama first, verify Hermes 3 runs, then launch Hermes Agent through Ollama.

TERMINOLOGY NOTE

whichllm is not vLLM. whichllm recommends models. vLLM serves models. Ollama runs models locally. Hermes Agent is launched through Ollama using a model such as hermes3.

1. Install whichllm on Windows, macOS, and Linux

whichllm requires Python 3.11 or later. The fastest one-time command is uvx. For repeat use, install it permanently with uv tool install, Homebrew, or pip.

Windows

Run these commands in PowerShell. Confirm Python 3.11 or later is installed first.

py -3.11 --version

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

uvx whichllm uv tool install whichllm whichllm –version whichllm hardware

macOS

Homebrew is the cleanest permanent install on macOS. uvx is useful when you only want to run it once.

# Homebrew install
brew install andyyyy64/whichllm/whichllm

whichllm –version whichllm hardware

# One-time run without permanent install
curl -LsSf https://astral.sh/uv/install.sh | sh

uvx whichllm

Linux

Use uv for a clean user-level install, or pip if your Python environment is already managed. python3 –version

curl -LsSf https://astral.sh/uv/install.sh | sh

exec $SHELL

uvx whichllm uv tool install whichllm whichllm hardware Alternative pip install

python -m pip install --upgrade pip
python -m pip install --upgrade whichllm

whichllm –version Basic verification whichllm whichllm –profile general –top 5 –status

whichllm –gpu "RTX 4090" –top 5

whichllm –cpu-only –top 5

1. Use whichllm to Choose a Local Writer Model

For a writing assistant, start with the general profile. Use coding, math, or vision only when that task type is the main job. Need Command Result General writing whichllm –profile general –top 5 –status Best general-purpose local models that fit your machine. Coding help whichllm –profile coding –top 5 –status Coding-specialized options.

CPU-only use

whichllm –cpu-only –top 5 Models ranked for RAM and CPU execution. Hardware planning

whichllm –gpu "RTX 4090" –top 5

What a target GPU would likely run. Machine-readable output whichllm –top 1 –json A JSON record that scripts can parse.

Save the top model ID

On macOS or Linux, install jq if you want to parse JSON in shell scripts. whichllm –profile general –top 1 –json | jq -r '.models[0].model_id' whichllm –profile coding –top 1 –json | jq -r '.models[0].model_id'

In PowerShell, ConvertFrom-Json gives the same result. $result = whichllm –profile general –top 1 –json | ConvertFrom-Json $result.models[0].model_id

OLLAMA NAME MAPPING

whichllm returns Hugging Face model IDs. Ollama model names do not always match those IDs. If the top recommendation is not in the Ollama library, choose the closest Ollama model by family and size, or run the Hugging Face model through whichllm run or vLLM. Run immediately with whichllm

# Auto-pick the best model for the current machine and start chat

whichllm run

# Run a specific GGUF-style match

whichllm run "qwen 2.5 1.5b gguf"

# Print a Python loading snippet instead of running chat

whichllm snippet "qwen 7b"

1. Install vLLM on Windows, macOS, and Linux

SUPPORT BOUNDARY

The vLLM quickstart lists Linux as the primary OS and Python 3.10 to 3.13. vLLM does not support native Windows. Use WSL2 on Windows. For Apple Silicon, use the separate vLLM-Metal plugin for GPU acceleration.

Linux with NVIDIA CUDA

Use this path for most NVIDIA GPU servers and workstations.

curl -LsSf https://astral.sh/uv/install.sh | sh

exec $SHELL

uv venv –python 3.12 –seed

source .venv/bin/activate

uv pip install vllm –torch-backend=auto

vllm serve Qwen/Qwen3-0.6B –host 0.0.0.0 –port 8000

Linux with AMD ROCm

Use this only on supported ROCm hardware and drivers.

curl -LsSf https://astral.sh/uv/install.sh | sh

exec $SHELL

uv venv –python 3.12 –seed

source .venv/bin/activate

uv pip install vllm –extra-index-url https://wheels.vllm.ai/rocm/

vllm serve Qwen/Qwen3-0.6B –host 0.0.0.0 –port 8000

Windows through WSL2

Run vLLM inside Ubuntu on WSL2. Native Windows vLLM is not the supported path.

# PowerShell as Administrator

wsl –install -d Ubuntu-24.04 wsl

# Inside Ubuntu

sudo apt update sudo apt install -y curl git

curl -LsSf https://astral.sh/uv/install.sh | sh

exec $SHELL

uv venv –python 3.12 –seed

source .venv/bin/activate

uv pip install vllm –torch-backend=auto

vllm serve Qwen/Qwen3-0.6B –host 0.0.0.0 –port 8000

macOS Apple Silicon with vLLM-Metal

Use vLLM-Metal for Apple Silicon GPU acceleration. It uses MLX-optimized models.

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
source ~/.venv-vllm-metal/bin/activate

vllm serve

macOS CPU source build

Use this only when you specifically need the CPU build on macOS. Install Xcode Command Line Tools first. xcode-select –install

git clone https://github.com/vllm-project/vllm.git
cd vllm

uv pip install -r requirements/cpu.txt –index-strategy unsafe-best-match uv pip install -e .

Verify the vLLM API

curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \

-d '{

"model": "Qwen/Qwen3-0.6B",

"messages": [{"role": "user", "content": "Write a one-sentence setup check."}] }'

1. Install Ollama on Windows, macOS, and Linux

Ollama is the easiest path for local model pulls, local chat, the localhost API, and the Hermes Agent launch

command. System Install command or path Requirement

Windows

irm https://ollama.com/install.ps1 | iex

Windows 10 or later

macOS

curl -fsSL https://ollama.com/install.sh | sh, or download the app

macOS 14 Sonoma or later

Linux

curl -fsSL https://ollama.com/install.sh | sh

Modern x86_64 or ARM64 Linux

Verify Ollama

ollama --version
ollama list
ollama run llama3.2:3b

Use the local Ollama API

curl http://localhost:11434/api/chat \

-d '{ "model": "llama3.2:3b",

"messages": [{"role": "user", "content": "Confirm Ollama is working."}]

}'

OPERATIONAL NOTE

On desktop systems, the Ollama app or service must be running before API calls work. If localhost:11434 refuses connections, start Ollama or run ollama serve in a terminal.

1. Set Up Hermes 3 and Hermes Agent with Ollama

Ollama lists Hermes 3 as the latest Hermes model family by Nous Research. It also lists Hermes Agent as an

application launch target.

Ollama tag

Download size Context When to choose it hermes3:3b

0 GB

128K Low-memory laptops or quick tests. hermes3:8b / latest

7 GB

128K Default choice for most local writing and agent tests. hermes3:70b

GB

128K High-memory workstation or server. hermes3:405b

GB

128K Specialized multi-GPU or server-class setup.

MEMORY NOTE

The size shown by Ollama is the model download size, not the full runtime memory budget. If a model fails to load, choose a smaller Hermes 3 tag first.

Pull and test Hermes 3

ollama pull hermes3:8b
ollama run hermes3:8b

Launch Hermes Agent

# Default Hermes 3 model
ollama launch hermes --model hermes3
# Smaller model for low-memory machines
ollama launch hermes --model hermes3:3b

Verify Hermes 3 through the API

curl http://localhost:11434/api/chat \

-d '{ "model": "hermes3:8b", "messages": [{"role": "user", "content": "Create a three-step writing plan."}] }'

1. Operating Playbooks

Path A: recommend and chat immediately

whichllm –profile general –top 5 –status whichllm run

Path B: recommend with whichllm, run in Ollama

1 Run whichllm and note the model family, parameter size, and quantization. 2 Search the Ollama library for the closest matching model name. 3 Pull and run the Ollama model. If no close Ollama match exists, use whichllm run or vLLM instead. whichllm –profile general –top 1 –json | jq -r '.models[0].model_id'

ollama pull hermes3:8b
ollama run hermes3:8b

Path C: serve through vLLM

whichllm –gpu "RTX 4090" –top 5

vllm serve Qwen/Qwen3-0.6B –host 0.0.0.0 –port 8000

curl http://localhost:8000/v1/models

Path D: Hermes Agent through Ollama

ollama pull hermes3:8b
ollama launch hermes --model hermes3

1. Troubleshooting Checklist

Symptom Likely cause Fix

vLLM install fails on Windows

Native Windows is not supported by vLLM.

Install WSL2 and run the Linux commands

inside Ubuntu.

vLLM cannot see the GPU

Driver, CUDA, ROCm, or WSL GPU

passthrough is missing. Verify nvidia-smi or rocminfo before installing vLLM. whichllm run fails uv is not installed or not in PATH. Install uv and open a new terminal.

whichllm returns Hugging Face

ID but Ollama cannot run it

Ollama names differ from Hugging Face

IDs. Map by model family and size in the Ollama library, or use whichllm run/vLLM.

Ollama API connection refused
Ollama is not running.

Start the app or run ollama serve, then retry localhost:11434.

Hermes Agent launch fails
Ollama version or launch integration is

unavailable.

Update Ollama, run ollama launch –help, and

verify ollama run hermes3 works. Model runs out of memory Selected model is too large for RAM/VRAM. Use hermes3:3b, lower quantization, or a smaller whichllm recommendation. Slow token generation CPU fallback or partial GPU offload.

Choose a smaller model, use GGUF/Ollama,

or move serving to a stronger GPU.

1. Sources Checked

These sources were checked on May 20, 2026 while preparing this guide.

whichllm README: https://github.com/Andyyyy64/whichllm

whichllm CLI reference: https://github.com/Andyyyy64/whichllm/blob/main/docs/cli.md

whichllm run and snippet docs: https://github.com/Andyyyy64/whichllm/blob/main/docs/run-snippet.md

vLLM quickstart: https://docs.vllm.ai/en/latest/getting_started/quickstart/

vLLM GPU installation: https://docs.vllm.ai/en/latest/getting_started/installation/gpu/

vLLM CPU installation: https://docs.vllm.ai/en/latest/getting_started/installation/cpu/

vLLM-Metal README: https://github.com/vllm-project/vllm-metal

Ollama quickstart: https://docs.ollama.com/quickstart
Ollama downloads: https://ollama.com/download
Ollama Hermes 3 library page: https://ollama.com/library/hermes3