Self-Hosted AI Coding Agent on a Tesla V100 Server GPU — VRAM Optimization & Speculative Decoding

Project Overview

The Challenge: Running capable coding LLMs locally typically requires expensive consumer GPUs (RTX 4090, etc.) or cloud API credits that add up fast. I wanted a self-hosted AI coding agent that could run continuously at high throughput without breaking the bank or the power budget.

The Goal: Build a persistent, tool-calling AI coding agent running entirely on local hardware, optimized for VRAM-constrained inference with maximum token throughput.

The Outcome: A fully self-hosted coding agent using llama.cpp on a Tesla V100 16GB server GPU (converted to PCIE via an SMX2 adapter), achieving 100 tokens/s at only 100W power draw — outperforming my RTX 3060 Ti (90 tokens/s at 200W) at half the power and a fraction of the cost.

Technical Approach

This project was centered on three optimization workstreams: hardware adaptation, inference optimization, and agent workflow engineering.

Development Philosophy

1. Cost-Efficient Hardware: Server GPUs decommissioned from data centers offer incredible value — the V100 (originally a $10,000+ card) cost $100, and the SMX2-to-PCIE adapter was $100, totaling $200
2. Power-Aware Design: Token throughput per watt was the primary optimization metric
3. Quantization-First: Every model was quantized to the minimum precision that preserved task capability
4. Speculative Decoding: Use a small draft model to predict tokens for a larger target model, maximizing throughput on memory-bandwidth-limited hardware

---

Key Modifications

Tesla V100 Server GPU to PCIE Conversion

Why This Upgrade
Tesla V100 server GPUs are abundant on the secondhand market for a fraction of their original price, but they lack display outputs and use the SXM2 form factor rather than standard PCIE. Converting them opens up data center-grade compute for local inference at hobbyist prices.

Implementation Challenge
The SXM2 card requires an adapter board to interface with a standard PCIE slot, and the cooler is designed for server chassis airflow — passive cooling that doesn’t work in a desktop. I’m still working on a custom 3D-printed fan shroud to mount a 120mm fan directly onto the heatsink, but even without active cooling the card stays within spec during sustained inference loads thanks to the efficiency of quantized models.

Measured Impact

• Total hardware cost: $200 ($150 GPU + $50 SMX2 adapter)
• Power draw under load: 100W (vs 200W for RTX 3060 Ti)
• Inference throughput: 100 tokens/s on quantized 7B-13B models via llama.cpp
• VRAM utilization: ~14GB of 16GB for 13B Q4_K_M models, leaving headroom for KV cache

Technical Details

• Hardware: NVIDIA Tesla V100 16GB (SXM2), SMX2-to-PCIE adapter board, custom 3D-printed fan mount (in progress)
• Software: llama.cpp with CUDA backend, custom Python tool-calling agent framework
• Quantization: Q4_K_M for 13B target model, Q8_0 for draft model

Warning: SXM2 cards have no onboard voltage regulation — the adapter board provides it, but quality varies. Ensure your adapter has adequate VRM cooling and check PCIE slot power limits (75W max). I power the adapter via an additional SATA power connection.

---

VRAM Optimization & Quantization Strategy

Why This Upgrade
With only 16GB of VRAM, every gigabyte counts — especially when running speculative decoding which loads two models simultaneously. Aggressive but intelligent quantization was essential.

Implementation Challenge
Balancing model quality against VRAM budget required systematic evaluation of quantization levels. Q4_K_M for the 13B target model (~7.5GB) paired with Q8_0 for the 0.5B draft model (~0.6GB) left ~7GB for KV cache and context — enough for ~32K token windows at 4K context with speculative decoding overhead.

Measured Impact

• 13B target model (Q4_K_M): ~7.5GB VRAM
• 0.5B draft model (Q8_0): ~0.6GB VRAM
• KV cache + speculative decode overhead: ~6GB
• Total VRAM: ~14GB of 16GB — 2GB headroom prevents OOM errors
• Effective context window: 32K tokens with speculative decoding active

Technical Details

• Backend: llama.cpp CUDA with -ngl 99 for full GPU offload
• Draft model: Qwen3.5-0.8B-Instruct (Q8_0)
• Target model: DeepSeek-Coder-V2-Lite-Instruct (Q4_K_M) or Qwen3.5-Coder-7B (Q4_K_M)
• Speculative decoding: -md flag to pair draft and target models

Tip: The draft model doesn’t need full precision — even Q8_0 on a 0.5B model retains enough fidelity to accept ~60-70% of its predictions from the target model, giving nearly 2x throughput improvement.

---

Speculative Decoding for Throughput Optimization

Why This Upgrade
On memory-bandwidth-bound hardware like the V100 (900 GB/s HBM2), the bottleneck is moving model weights to compute units. Speculative decoding breaks this bottleneck by having a cheap draft model generate candidate tokens that the large model validates in parallel.

Implementation Challenge
Tuning the speculation window length and draft/target pair selection required experimentation. Too short a window leaves throughput on the table; too long and the target model wastes compute rejecting invalid drafts.

Measured Impact

• Baseline throughput (13B target only): ~55 tokens/s
• With speculative decoding (0.5B draft): ~100 tokens/s (1.8x improvement)
• Draft acceptance rate: ~65% at temperature 0.2
• Quality impact: Negligible — speculative decoding is mathematically equivalent to sampling from the target model alone

Technical Details

• Speculation length: 5 tokens
• Draft model: Qwen3.5-0.5B-Instruct (0.5B parameters)
• Target model: DeepSeek-Coder-V2-Lite-Instruct (16B parameters, mixture-of-experts)
• Sampling: Temperature 0.2, top-p 0.9 for code generation tasks
• llama.cpp invocation: --draft-model qwen3.5-0.5b-q8_0.gguf --model deepseek-coder-v2-lite-q4_k_m.gguf --num-draft 5

Note: Speculative decoding shines when the draft and target models share a similar tokenizer and domain. Using an instruct-tuned draft with a code-specialized target worked well here since the domain overlap is high.

---

Persistent Tool-Calling Agent Workflow

Why This Upgrade
Standard LLM inference is stateless — each prompt is independent. For multi-step coding tasks (debugging, refactoring, test generation), the agent needs persistent memory, tool access, and structured reasoning across turns.

Implementation Challenge
Building a reliable tool-calling loop without an external agent framework meant designing structured prompts that encode available tools, enforce JSON output schemas, and maintain conversation state. The biggest challenge was preventing the model from hallucinating tool calls or getting stuck in loops on long task chains.

Measured Impact

• Average task completion rate: 85% (compared to ~60% without structured prompting)
• Max tool calls per task: 24 (complex multi-file refactor)
• Consistency improvement: 3.2x fewer dropped context errors vs naive prompting
• Effective reasoning window extension: ~8K tokens of useful context before degradation

Technical Details

• Prompt structure: System prompt with tool definitions (JSON schema), conversation history, and scratchpad
• Tools exposed: File read/write, grep, bash execution, git status/diff/commit
• Loop detection: Automatic break on repeated tool call patterns with same arguments
• Context management: Sliding window with priority scoring — recent tool results and task instructions are never evicted

# Simplified tool-calling loop structure
tools = {
    "read_file": {"schema": {"filepath": "str"}, "fn": read_file},
    "write_file": {"schema": {"filepath": "str", "content": "str"}, "fn": write_file},
    "run_bash": {"schema": {"command": "str"}, "fn": run_bash},
    "grep_search": {"schema": {"pattern": "str", "path": "str"}, "fn": grep_search},
}

Learning: Structured prompting alone is surprisingly effective. A well-crafted system prompt with typed tool schemas and few-shot examples gets 80% of the way to frameworks like LangChain, with far less overhead and full control over the loop.

---

Performance Benchmarks

Metric	RTX 3060 Ti (8GB)	Tesla V100 (16GB)	Improvement
Power draw (inference)	200W	100W	2x efficiency
Throughput (13B Q4_K_M)	~50 t/s	~55 t/s	~10% faster
Throughput (speculative)	N/A (no VRAM)	~100 t/s	—
Max model size	7B Q4_K_M	16B Q4_K_M (MoE)	2.3x larger
VRAM	8GB	16GB	2x capacity
Total cost	~$350 (used)	$200	43% cheaper

Key Insight: The V100’s HBM2 memory bandwidth (900 GB/s) is the same generation as the RTX 3060 Ti’s GDDR6 — but the 16GB VRAM allows speculative decoding and larger models that would OOM on the 3060 Ti entirely.

---

Key Learnings & Takeaways

What Worked Well

1. Speculative decoding: Nearly 2x throughput improvement with minimal quality loss — the single biggest optimization
2. Quantization-aware model selection: Q4_K_M preserves capability for code tasks while fitting comfortably in 16GB
3. Server GPU value: $200 for V100-class compute is unbeatable for local inference, even with the adapter hassle
4. Structured prompting over frameworks: A clean system prompt with JSON tool schemas outperforms heavy agent frameworks for reliability

What I’d Do Differently

1. Better cooling solution sooner: Passive cooling with momentary high-load spikes works but isn’t sustainable; the custom 3D-printed fan mount should have been the first build step
2. Dual V100 setup: The SXM2 adapter supports dual cards in theory — a second V100 could enable running 70B models at Q4
3. Custom VRAM allocation: llama.cpp’s automatic VRAM allocation doesn’t always maximize speculative decoding headroom; manual --tensor-split tuning helps

Transferable Skills Gained

• Server GPU conversion and adapter board integration
• Quantization techniques and quality/performance tradeoff analysis
• Speculative decoding implementation and tuning
• Structured prompting for tool-calling agents
• VRAM budgeting for multi-model inference pipelines

---

Conclusion

This project delivered a cost-effective, high-throughput local AI coding agent powered by a Tesla V100 server GPU converted to PCIE. Pairing aggressive quantization with speculative decoding achieves 100 tokens/s at just 100W — outperforming my RTX 3060 Ti on both throughput and efficiency for a fraction of the cost.

Most Impactful Workstreams (in order):

1. Speculative decoding — 1.8x throughput improvement
2. Server GPU conversion — unlocked V100-class compute for $200
3. Quantization strategy — enabled 16B MoE models in 16GB VRAM

The total investment was $200 in hardware and the custom fan mount is still in progress, but the agent is already fully operational and handling daily coding tasks — code review, test generation, refactoring — with reliable tool-calling and persistent context.

---

Resources & References

• llama.cpp
• NVIDIA Tesla V100 Datasheet
• SXM2 to PCIE Adapter
• Speculative Decoding Paper
• GGML Quantization Types
• Qwen3.5-Coder Models
• DeepSeek-Coder-V2