LLM Quantization: GPU Memory Strategy for Enterprise AI Teams
This article is a companion to the existing overview of speculative decoding, quantization, and distillation tradeoffs. That piece compared three optimization paths. This draft goes deeper on quantization because it is often the fastest route from an impressive lab model to a deployable model that fits the GPU budget.
For CTOs and AI engineering leaders in Southeast Asia, quantization belongs in the infrastructure design conversation. It changes the memory footprint of the model, the practical batch size, the number of concurrent sessions a GPU can support, and the cost of private or on-premise deployment. It also changes the risk surface: a lower-bit model that passes a generic benchmark still fails production when retrieval, extraction, tool calling, or long-context behavior regresses.
What LLM quantization in production AI is
LLM quantization reduces the numerical precision used to store or compute model values. A full-precision serving path often uses FP16 or BF16. A quantized path uses formats such as FP8, INT8, or INT4 for some combination of weights, activations, and KV cache. vLLM describes quantization as a precision tradeoff that creates a smaller memory footprint and broader device fit.
The practical point is simple: every token generated by an autoregressive model is constrained by memory traffic as well as compute. Weights must be loaded. Activations must move through matrix operations. Long conversations and RAG payloads grow the KV cache. Quantization reduces one or more of those memory burdens, which gives the serving layer more room for larger models, longer contexts, higher concurrency, or fewer GPUs.
The method is mature enough to be operationally useful, but it is not one switch. Hugging Face separates common choices by use case: bitsandbytes for ease of use, GPTQModel or AWQ for calibrated 4-bit inference, compressed-tensors for FP8 and sparse formats, and torchao for PyTorch-native paths. Its own guidance says teams should benchmark on the specific task and hardware. That sentence is the production rule.
Why quantization reduce GPU cost and memory pressure
The first cost driver is weight memory. A 70B-parameter model in BF16 needs roughly 140 GB just for weights before runtime overhead, KV cache, framework buffers, and serving headroom. Weight-only 8-bit quantization brings the raw weight storage close to half. Weight-only 4-bit quantization brings it close to one quarter. The real footprint is higher because scales, metadata, kernels, and cache still exist, but the direction is decisive.
The second cost driver is activation and matrix math. SmoothQuant targets W8A8 quantization, meaning 8-bit weights and 8-bit activations. The ICML 2023 paper reported up to 1.56x speedup and 2x memory reduction in its tested LLM settings while preserving accuracy under its experimental conditions. The important mechanism is activation outlier handling: SmoothQuant moves quantization difficulty from activations to weights using an offline transformation, because activations are harder to quantize cleanly.
The third cost driver is the KV cache. In long-context assistants, the cache becomes a large and growing memory allocation as sessions accumulate tokens. vLLM documents FP8 KV cache as a way to significantly reduce KV-cache memory footprint, which increases the number of tokens that fit in memory and improves throughput when the workload is cache-bound.
A CTO should read those three cost drivers separately. Weight quantization helps fit the model. Activation quantization helps the compute path. KV-cache quantization helps long sessions and concurrency. A cost model that blends all three into one percentage hides the actual bottleneck.
Which quantization formats matter for enterprise serving
The useful production map starts with the hardware, then the serving stack, then the task. A format that works in a notebook is not automatically efficient on the target GPU fleet.
NVIDIA TensorRT documents the low-level behavior of INT8, FP8, INT4, and NVFP4 quantization schemes, including scale handling and the fact that TensorRT supports INT4 for weight quantization. TensorRT-LLM also exposes recipes for FP8, INT4 AWQ, INT8 SmoothQuant, KV-cache FP8, and mixed auto-quantization paths through its quantization toolkit.
vLLM's hardware matrix makes this concrete. Its current documentation lists support across AWQ, GPTQ, INT8 W8A8, FP8 W8A8, bitsandbytes, GGUF, and quantized KV cache, with support varying by Volta, Turing, Ampere, Ada, Hopper, AMD GPU, Intel GPU, and x86 CPU. The production question is therefore not simply whether the model can be quantized. The question is whether the chosen quantization method is fast on the exact serving path.
How AWQ, GPTQ, SmoothQuant, and LLM.int8 differs
LLM.int8() addresses the outlier problem directly. The NeurIPS 2022 paper describes a mixed-precision decomposition that keeps emergent outlier dimensions in 16-bit while more than 99.9% of values are multiplied in 8-bit. That is the right mental model for many enterprise workloads: most values tolerate lower precision, while a small sensitive subset needs protection.
SmoothQuant generalizes the outlier lesson for W8A8 deployment. It smooths activation outliers by shifting difficulty into weights before quantization. This matters for serving because weight-only compression lowers memory, while activation quantization changes the runtime compute path. Teams chasing lower cost per token need to test both.
GPTQ is a one-shot post-training weight quantization method. The ICLR 2023 paper reports that GPTQ quantized 175B-parameter models to 3 or 4 bits per weight in approximately four GPU hours with negligible degradation in its tests, and reported end-to-end inference speedups over FP16 on A100 and A6000 hardware.
AWQ protects salient weights based on activations. The AWQ paper argues that protecting only 1% of salient weights can greatly reduce quantization error, and reports strong 3-bit and 4-bit deployment results. In practice, AWQ and GPTQ are common candidates when the problem is fitting a larger model into a smaller GPU envelope.
Where the quantization hype break down
Quantization breaks down when teams measure average benchmark score and ignore the business task. Enterprise regressions often appear in narrow places: a wrong invoice field, a missed contradiction in a policy document, a weaker Bahasa Indonesia response, a malformed tool-call argument, or a longer chain of reasoning that quietly becomes less stable.
KV-cache quantization is a clear example. In April 2026, vLLM maintainers described long-context FP8 KV-cache work where earlier Hopper tests on a 128k needle-in-a-haystack task dropped from 91% BF16 accuracy to 13% before kernel fixes. After fixes, the same article positioned FP8 KV cache as a default starting point for many long-context deployments, while still naming exceptions around prefill-heavy workloads, head dimensions, hybrid attention, and calibration needs.
That is the honest lesson. Quantization is powerful because many inference workloads are memory-bound. It is risky because the failure modes are workload-specific. The rollout gate must include real prompts, real retrieved context, real documents, real tool outputs, and the error examples that matter to the business.
What CTOs should test before approving quantization
The release test should compare the baseline model and every quantized candidate under the same production trace. A useful gate includes:
• Memory: model weights, KV cache, peak VRAM, fragmentation, and safe headroom under burst traffic.
• Latency: time to first token, inter-token latency, full response latency, and p95 or p99 variance.
• Throughput: tokens per second under realistic concurrency, batching, and context lengths.
• Quality: extraction accuracy, groundedness, reasoning depth, refusal behavior, citation faithfulness, and Bahasa Indonesia/domain-language handling.
• Reliability: rollback path, model-version tagging, drift monitoring, calibration data lineage, and incident thresholds.
• Economics: GPU count, utilization, autoscaling profile, serving framework constraints, and cloud versus private deployment cost.
The pass/fail threshold should be task-specific. A 4-bit model that reduces memory by a large margin is attractive for internal summarization. The same model needs stricter gates for trade finance verification, medical document processing, HR policy interpretation, or regulated customer workflows. The deeper the downstream action, the tighter the evaluation harness.
What the practical decision framework is
1. Start with the bottleneck. Use quantization first when GPU memory, hardware placement, or serving cost blocks deployment.
2. Separate weights, activations, and KV cache. A single quantization label hides three different memory problems.
3. Choose the format around the hardware. FP8, INT8, and INT4 depend on kernels, GPU generation, framework support, and batch shape.
4. Calibrate with production-like data. Calibration should include domain language, retrieval payloads, long context, and known failure cases.
5. Ship behind a quality gate. Compare baseline and quantized candidates on task-level metrics before changing the default path.
6. Keep the rollback route alive. Some workflows should stay on BF16/FP16 or route hard cases to a higher-precision model.
FAQ
Is quantization the best first optimization for enterprise AI serving?
Quantization is the best first test when the constraint is GPU memory, cost per request, or private deployment fit. Speculative decoding is stronger for token latency when quality is already acceptable. Distillation is stronger when the task is narrow, stable, and worth training into a smaller model.
Does INT4 quantization reduce model quality?
INT4 quantization increases regression risk, especially for reasoning, extraction, code, long-context retrieval, and domain-specific language. Good AWQ or GPTQ checkpoints often perform well, but enterprise teams still need baseline comparisons on real prompts and documents before approving rollout.
When should a team use FP8 instead of INT8 or INT4?
FP8 is attractive when the target hardware and serving stack support efficient FP8 kernels, especially for newer NVIDIA or AMD paths and KV-cache strategies. INT8 remains useful for broad accuracy retention. INT4 is mainly a weight-memory strategy when model placement is the dominant constraint.
Why does KV-cache quantization matter for RAG?
RAG workloads often send long retrieved contexts into the model. The KV cache grows with context and active sessions, so it becomes a concurrency and cost constraint. FP8 KV-cache quantization can fit more tokens in memory, but it needs long-context accuracy testing.
Can quantization support private or on-premise AI deployment?
Yes. Quantization can make private deployment practical by reducing the GPU memory required for a chosen model and workload. The decision still depends on the client's hardware, data-control requirements, serving framework, and evaluation results under production-like traffic.
How should enterprise teams evaluate quantized models?
Evaluate quantized models against the baseline using real prompts, retrieved context, tool outputs, document samples, concurrency patterns, and known failure examples. The goal is measured production fit: lower cost and memory use without weaker groundedness, extraction accuracy, or auditability.
