NemoClaw + vLLM: Local Inference Performance Optimization on RTX and DGX

NemoClaw is NVIDIA’s enterprise security wrapper for agentic AI — the sandboxed runtime environment that adds kernel-level isolation, a YAML policy engine, and a privacy router to the open-source OpenClaw agent framework. When NemoClaw connects to a local vLLM inference server instead of a cloud API, every token of reasoning stays on hardware you physically control. No API calls leave the machine. No data crosses a network boundary. This is the configuration that turns NemoClaw from a security layer into a fully sovereign AI agent stack.

vLLM is the open-source inference engine that NemoClaw uses natively through its vllm-local provider. It is not a third-party integration or a community plugin. NVIDIA built vLLM support directly into NemoClaw’s provider architecture because local inference is the only way to deliver the data privacy guarantees that regulated enterprises require. This article covers the complete technical path: hardware selection, vLLM installation, NemoClaw provider configuration, PagedAttention tuning for concurrent agent workloads, and the gateway VRAM management system that prevents idle inference servers from consuming GPU memory.

For the hardware decision framework behind these configurations, see our NemoClaw Hardware Guide. For the architecture that connects the privacy router to local inference, see our NemoClaw Architecture Deep Dive. For pricing on managed deployment of this stack, see our Pricing page.

NemoClaw’s privacy router can route requests to cloud LLM endpoints while stripping PII, but cloud routing always involves a trust boundary. The cloud provider sees the sanitized prompt. The network path between your host and the API endpoint is encrypted but traverses infrastructure you do not own. Compliance auditors for HIPAA, ITAR, and classified workloads will not accept this architecture regardless of how thorough the PII stripping is.

Local vLLM inference eliminates the network trust boundary entirely. The inference engine runs on the same physical machine — or the same air-gapped rack — as the NemoClaw sandbox. The privacy router’s role shifts from “strip PII before sending to cloud” to “verify that no request accidentally escapes to an external endpoint.” This is a fundamentally stronger security posture because the failure mode changes from “data leaked with PII removed” to “data never left the machine.”

NVIDIA’s Nemotron 3 Super family is purpose-built for NemoClaw deployments. Two variants are currently active: the 120B parameter model for production workloads and the 12B parameter model for development, testing, and edge deployments. Both use a hybrid Mixture-of-Experts (MoE) architecture that activates only a subset of parameters per token, delivering the quality of a dense model at a fraction of the compute cost.

The 120B model supports a 1-million-token context window. This is not a marketing number — it is the operational context that allows an agent to ingest an entire codebase, a complete regulatory filing, or months of email history in a single session without truncation. For agentic workflows where the agent needs to reference earlier actions, tool outputs, and accumulated context across dozens of steps, the 1M context window eliminates the retrieval-augmented generation workarounds that smaller context models require.

The hybrid MoE architecture means that while the model has 120B total parameters, each forward pass activates roughly 40B parameters. This is why a model that would normally require 240GB of VRAM in FP16 can run on significantly less memory with quantization — and why NVFP4 on Blackwell GPUs delivers 4x the throughput compared to FP8 on the previous-generation H100.

NVIDIA’s Blackwell architecture (B200, GB200, GB10 in DGX Spark) introduces native NVFP4 — a 4-bit floating-point format with hardware-level support in the Tensor Cores. Previous-generation H100 GPUs top out at FP8 precision for inference. The H200 — NVIDIA’s highest-memory inference chip as of March 2026 — still uses Hopper architecture and lacks native NVFP4 support. The jump from FP8 to NVFP4 is not just a 2x memory reduction — Blackwell’s Tensor Cores are architecturally designed to deliver 4x higher throughput vs FP8 on H100 while maintaining accuracy.

For NemoClaw local inference, this means a single RTX 5090 (32GB GDDR7, Blackwell architecture) running Nemotron 3 Super 12B in NVFP4 can serve concurrent agent requests at throughput levels that previously required an H100 GPU costing 5–10x more. The DGX Spark with its GB10 Grace Blackwell SoC can run the full 120B model in NVFP4 with headroom for PagedAttention’s dynamic memory allocation.

Recommended Dev Spec for Nemotron Nano 30B

For teams working with the Nemotron Nano 30B model — the mid-range option between the 12B dev model and the 120B production model — the following hardware specification provides a smooth local development experience with vLLM. The official NVIDIA vLLM Recipes user guide documents these requirements for the Nemotron-3-Nano-30B model.

The vLLM blog published an official Nemotron 3 Super integration guide that covers the model download and server startup process. The following walkthrough adapts that guide for NemoClaw’s specific provider configuration and adds the gateway VRAM management that the official guide omits.

Step 1: Install vLLM with NVIDIA Extensions

# Install vLLM with CUDA 12.4 support
$ pip install vllm --extra-index-url https://download.pytorch.org/whl/cu124

# Verify GPU detection
$ python -c "import vllm; print(vllm.__version__)"
0.8.3

# Verify CUDA is available
$ python -c "import torch; print(torch.cuda.get_device_name(0))"
NVIDIA GeForce RTX 5090

Step 2: Download and Serve Nemotron 3 Super

# Start vLLM serving Nemotron 3 Super 12B with NVFP4 quantization
$ vllm serve nvidia/nemotron-3-super-12b \
    --quantization nvfp4 \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.85 \
    --enable-chunked-prefill \
    --port 8000 \
    --host 127.0.0.1

# For the 120B model on multi-GPU (DGX or 4x H100):
$ vllm serve nvidia/nemotron-3-super-120b \
    --quantization nvfp4 \
    --tensor-parallel-size 4 \
    --max-model-len 1048576 \
    --gpu-memory-utilization 0.90 \
    --enable-chunked-prefill \
    --port 8000 \
    --host 127.0.0.1

Step 3: Configure NemoClaw’s vllm-local Provider

providers:
  default: vllm-local
  vllm-local:
    endpoint: "http://127.0.0.1:8000/v1"
    model: "nvidia/nemotron-3-super-12b"
    max_tokens: 8192
    temperature: 0.1
    timeout_seconds: 120

privacy_router:
  mode: local-only
  block_external: true
  allowed_endpoints:
    - "127.0.0.1:8000"

Setting privacy_router.mode: local-only and block_external: true ensures that even if a future configuration change adds a cloud provider, the privacy router will block any request that attempts to leave the machine. The allowed_endpoints list is a whitelist — only the local vLLM server on port 8000 will receive inference requests.

Step 4: Verify End-to-End Local Inference

# Test that NemoClaw is using the local vLLM provider
$ nemoclaw provider status
Active Provider: vllm-local
Endpoint: http://127.0.0.1:8000/v1
Model: nvidia/nemotron-3-super-12b
Status: CONNECTED
Privacy Router: local-only (external blocked)

# Send a test prompt through the sandboxed agent
$ nemoclaw run --prompt "List the files in /tmp" --dry-run
[sandbox] Agent would execute: ls /tmp
[provider] Request routed to: 127.0.0.1:8000 (local)
[privacy] External endpoint check: PASS (no external calls)

# Verify no network traffic leaves the host
$ ss -tnp | grep 8000
ESTAB  0  0  127.0.0.1:52340  127.0.0.1:8000  users:(("nemoclaw",pid=12345))

The default vLLM configuration is optimized for general-purpose inference — chatbots, single-user completions, batch processing. NemoClaw agent workloads have a different profile: multiple concurrent sessions, each with long context windows, issuing tool calls that generate bursts of short requests followed by idle periods. Tuning PagedAttention for this profile requires adjusting three parameters.

$ vllm serve nvidia/nemotron-3-super-12b \
    --quantization nvfp4 \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.85 \
    --enable-chunked-prefill \
    --max-num-seqs 16 \
    --max-paddings 512 \
    --swap-space 8 \
    --port 8000 \
    --host 127.0.0.1

# --max-num-seqs 16    : Allow up to 16 concurrent agent sessions
# --max-paddings 512   : Reduce wasted memory from padding
# --swap-space 8       : 8GB CPU swap for KV-cache overflow

--max-num-seqs 16 sets the maximum number of sequences (agent sessions) that vLLM will process concurrently. The default is 256, which is appropriate for chatbot workloads with short contexts but wastes memory pre-allocation for agent workloads where each session uses a much larger context window. Setting this to 16 tells PagedAttention to allocate KV-cache capacity for 16 long-context sessions rather than 256 short ones.

--swap-space 8 enables KV-cache swapping to CPU memory. When all GPU VRAM is consumed by active sessions, PagedAttention will swap the KV-cache of idle sessions to CPU RAM rather than rejecting new requests. This is essential for NemoClaw because agent sessions frequently pause while waiting for tool execution results — those paused sessions should release GPU memory for active sessions.

A running vLLM server with Nemotron 3 Super 12B loaded consumes approximately 12–14GB of VRAM even when idle — no requests, no active sessions, just the model weights sitting in GPU memory. On a workstation that also runs development tools, rendering software, or other GPU-accelerated workloads, this idle VRAM consumption is unacceptable.

NemoClaw’s gateway manages on-demand vLLM startup and shutdown to free VRAM when the inference server is idle. The gateway sits between NemoClaw’s provider interface and the vLLM server. When an agent session requires inference, the gateway starts the vLLM server, waits for the model to load, routes the request, and starts an idle timer. When no requests have been received for the configured idle period, the gateway sends a SIGTERM to the vLLM process, waits for graceful shutdown, and the VRAM is freed back to the system. This is particularly valuable on developer workstations where the same GPU serves both NemoClaw inference and other tasks like rendering, training, or IDE GPU-accelerated features.

gateway:
  enabled: true
  idle_timeout_minutes: 15
  startup_timeout_seconds: 120
  health_check_interval: 30
  vllm_command: "vllm serve nvidia/nemotron-3-super-12b --quantization nvfp4 --max-model-len 131072 --gpu-memory-utilization 0.85 --enable-chunked-prefill --port 8000 --host 127.0.0.1"
  shutdown_signal: SIGTERM
  graceful_shutdown_seconds: 30

The tradeoff is cold-start latency. Loading Nemotron 3 Super 12B in NVFP4 on an RTX 5090 takes approximately 15–25 seconds. For the 120B model on DGX, expect 45–90 seconds. If your agents run continuously throughout the workday, disable the gateway and keep vLLM running permanently. If your agents are used intermittently — a few sessions per hour — the gateway saves 12–14GB of VRAM during idle periods, which is significant on a 32GB GPU.