Overview
In this installment of our "Lab Notes" series, we're diving deep into the practicalities of running large language models (LLMs) on a setup that most hobbyists can afford. Specifically, I'll be benchmarking an Intel i5 14th gen CPU with 64GB DDR4 RAM and an NVIDIA RTX A4000 16GB GPU—a configuration that's both cost-effective and space-efficient.
Background
The RTX A4000 is a gem in the world of affordable workstations. At roughly $600-800 used, it offers impressive performance without requiring a beefy power supply or taking up too much space. This makes it an ideal choice for homelab enthusiasts who want to experiment with local AI models but don't have deep pockets.
How It Works
Methodology
The benchmarking method is straightforward and designed for reproducibility:
- LLM Server: I used Ollama's server with default settings, running models at their default quantization level (Q4_K_M class).
- Prompt Set: A fixed set of prompts—short, medium, and long—to ensure consistency across runs.
- Warming Up: One unrecorded warmup run followed by three recorded runs per prompt and model.
- Generation Settings: Each generation was capped at 256 tokens with a temperature of 0. The numbers reported are the median from Ollama's own counters (eval_count / eval_duration).
- Benchmark Script: A Python script that automates the process, which I'll share in a future post.
Results
Here’s what we found:
| Model | Generation Speed (Median Tokens/Second) |
|---|---|
| Qwen2.5:1.5B | 203 |
| Llama3.2:3B | 124 |
| Qwen2.5:7B | 63 |
Analysis
The results show a clear pattern: doubling the model size roughly halves the generation speed. This is expected because generation is memory bandwidth-bound, and producing each new token requires streaming essentially the whole model through memory.
Using a simple formula (448 GB/s / 4.8GB x ~0.65 real-world efficiency), we predicted that a 7B model at Q4 should run around 61 tokens/second, which is very close to our measured result of 63 tokens/second.
Results
Generation Speed
The benchmark results for different models are as follows:
- Qwen2.5:1.5B - 203 tok/s
- Llama3.2:3B - 124 tok/s
- Qwen2.5:7B - 63 tok/s
Model Scalability on the RTX A4000
Based on these results, we can draw some practical conclusions:
- 7B Class Models: These run at very comfortable interactive speeds (63 tokens/second is much faster than anyone reads).
- 14B Class Models: A 14B model at Q4 would fit with room for context and should land around 30 tokens/second, based on the same formula.
- 32B+ Class Models: These do not fit in 16GB of memory, requiring either quality-destroying quantizations or CPU offloading. A 70B model is out of the question with this card.
Lessons Learned
Prompt Processing Speed
A mistake worth admitting: Our first run recorded prompt processing speed, and those numbers were garbage. The initial runs showed slow prompt evaluation, but subsequent runs showed absurdly high numbers like 33,000 tokens/second. The cause was Ollama caching the KV state of identical prompts, leading to cache hits instead of full prompt processing.
Fix for Next Time
To ensure accurate measurements in future runs, we need to prepend a random nonce to each prompt so that every run pays full prompt processing cost. Until the script is fixed, we are publishing only the generation numbers.
Next Steps
- Fix Prompt Processing Measurement: Address the issue with prompt processing speed.
- Add 14B Model: Include results for a 14B model in the benchmark.
- Test Multi-GPU Setup: Determine if using a second cheap GPU is worth it for improved performance.
This series aims to provide real-world insights into running large language models on affordable hardware, helping homelab enthusiasts make informed decisions about their setups. Stay tuned for more updates!