Running FLUX.1 in 4-Bit: Nunchaku Brings High-Speed Quantized Diffusion to Diffusers

The Challenge: Memory Walls and Quantization Loss in DiTs

Modern text-to-image AI has shifted dramatically from classical UNet architectures to massive Diffusion Transformers (DiTs). While models like Black Forest Labs’ FLUX.1 offer unprecedented visual quality and prompt adherence, their computational footprint is prohibitive. In native 16-bit floating-point precision (FP16), running FLUX.1-dev requires approximately 24 to 32 GB of VRAM. For European studios, freelancers, and small tech firms, this meant relying either on expensive enterprise GPUs—such as the NVIDIA RTX 4090 (priced around €1,850 / $2,000) or enterprise-grade cloud instances—or paying per-generation API fees to US cloud providers.

Standard 4-bit quantization methods like NF4 (NormalFloat4) or INT4 weight-only quantization drastically reduce memory usage, but they introduce severe artifacts when applied to DiTs. Diffusion Transformers exhibit extreme activation outliers—small sets of hidden values that spike to high magnitudes during inference. When compressed into uniform 4-bit bins, these outliers cause severe quantization errors, resulting in blurry textures, distorted human limbs, and broken text rendering.

Enter SVDQuant and Nunchaku: How It Works

To eliminate this bottleneck, researchers introduced SVDQuant, a specialized quantization method co-developed by researchers at MIT, Tsinghua University, and Squeeze AI Lab. Instead of squeezing all weights directly into 4-bit precision, SVDQuant uses Singular Value Decomposition (SVD) to separate the outlier activations into a low-rank floating-point branch, while quantizing the remaining bulk matrix into 4-bit precision.

Complementing this architecture is Nunchaku, an inference engine tailored specifically for low-rank SVDQuant kernels. Nunchaku bypasses the overhead of traditional CUDA runtime kernels by executing the 4-bit matrix multiplication and the low-rank FP16 correction concurrently on NVIDIA Tensor Cores. The result is a system that retains nearly 99% of FP16 image quality while drastically reducing both memory bandwidth requirements and latency.

With its integration into Hugging Face’s diffusers Python library, developers can now deploy Nunchaku-quantized models natively with just a few lines of code, replacing heavy FP16 pipelines with optimized INT4 backends.

Benchmarking Performance and Memory Efficiency

To highlight the practical impact for developers, we compiled a comparison matrix showing how standard FP16 FLUX.1 compares against conventional NF4 quantization and Nunchaku SVDQuant 4-bit inference across hardware tiers:

Because Nunchaku optimizes kernel execution at the C++/CUDA level, memory bandwidth bottlenecks—which typically slow down 16-bit models on consumer GPUs—are greatly reduced. In our testing benchmarks across local test rigs in our AI Arena evaluation lab, reducing VRAM usage below 12 GB enables workflows that previously crashed due to Out-Of-Memory (OOM) errors on popular consumer graphics cards.

How to Use Nunchaku in Diffusers

Integrating Nunchaku into an existing Hugging Face pipeline requires minimal refactoring. Once the nunchaku package is installed alongside diffusers and accelerate, developers can load pre-quantized SVDQuant weights directly from the Hugging Face Hub:

from diffusers import FluxPipeline
import torch

# Load Nunchaku 4-bit FLUX.1 pipeline
pipe = FluxPipeline.from_pretrained(
    "mit-han-lab/svdquant-flux-1-dev",
    torch_dtype=torch.bfloat16
).to("cuda")

image = pipe(
    "A crisp architectural photo of a modern sustainable office building in Berlin, golden hour lighting",
    num_inference_steps=25,
    guidance_scale=3.5
).images[0]

image.save("berlin_office.png")

European Business Angle: Local Hosting and GDPR Compliance

The ability to run high-quality diffusion transformers locally on lower-cost hardware delivers significant operational advantages for European organizations:

• Strict GDPR & Data Privacy: Transferring client data, proprietary product designs, or personal images to third-party proprietary APIs based in non-EU jurisdictions carries regulatory compliance risks under European data privacy legislation. Running 4-bit open-weight models locally ensures end-to-end data control.
• Lower Operational Costs in EUR: Standard cloud API pricing for high-resolution images ranges between €0.02 and €0.08 per image. For a creative agency producing thousands of iterations monthly, hosting local instances on standard workstations equipped with an NVIDIA RTX 4070 or RTX 5060 Ti (costing €550 to €850 / $600 to $900) amortizes capital costs within months.
• Compliance with the EU AI Act: Open-weights models deployed on local infrastructure offer full transparency regarding system architecture, data processing, and security boundaries, easing documentation obligations under the EU AI Act framework.

Availability and Hardware Compatibility

The SVDQuant code and Nunchaku engine are open-source and globally available immediately on GitHub and the Hugging Face Hub. Currently, optimized CUDA kernels are supported on NVIDIA GPUs featuring Tensor Cores (Ada Lovelace, Ampere, and newer architectures, including RTX 30-series, 40-series, and 50-series GPUs). Apple Silicon (Metal) and AMD ROCm support are currently under active development by the community.