Qwen3-VL-8B: Significantly degraded visual understanding compared to LM Studio with same weights

[Hassan-A-K]

Description

Using the same model weights (lmstudio-community/Qwen3-VL-8B-Instruct-MLX-4bit), same image, same prompt, and same hardware (M5 Max 40C, 64GB), mlx-vlm produces dramatically worse visual understanding than LM Studio's mlx-engine.

Reproduction

Same engineering drawing with a visible As-Built stamp at 4 MP:

LM Studio (mlx-engine v1.4.0):

• Detects stamp, reads OCR text ("REDLINE AS-BUILT ☐ CHANGE ☑ NO CHANGE"), finds redlines
• features_detected: ["As-Built Stamp", "Change/No Change Checkbox", "Redline Annotations", "Material Substitution Notes"]
• as_built_stamp_detected: true

mlx-vlm (v0.4.0) via vllm-mlx and Bodega:

• Returns features_detected: []
• as_built_stamp_detected: false
• "No As-Built stamp or revision table found"

Accuracy on full dataset (53 files at 4 MP)

Server	Underlying Engine	Accuracy
LM Studio	mlx-lm + VisionAddOn	72.0%
vllm-mlx	mlx-vlm generate()	52-55%
Bodega	mlx-vlm (assumed)	41.5%

Investigation

We confirmed the image preprocessing is identical — pixel_values shape (15600, 1536) and image_grid_thw match exactly between HuggingFace processor and mlx-vlm's prepare_inputs(). The image reaches the model at full resolution in all cases.

The difference appears to be in how the text generation pipeline handles visual tokens after the vision encoder. LM Studio uses a "VisionAddOn" architecture where mlx-vlm is only used for vision embedding extraction, while text generation always goes through mlx-lm. This separation apparently produces significantly better results.

Specific areas that may differ:

1. Chat template application — how vision tokens (<|vision_start|>, <|image_pad|>, <|vision_end|>) are placed in the prompt
2. Deepstack visual embedding handling — Qwen3-VL injects intermediate vision features into early language model layers
3. Sampling pipeline — mlx-lm has more mature sampling (logits processors, etc.)

Environment

• mlx-vlm: 0.4.0
• mlx: 0.31.1
• transformers: 5.0.0rc3
• Hardware: Apple M5 Max 40C, 64GB
• Model: lmstudio-community/Qwen3-VL-8B-Instruct-MLX-4bit
• macOS 26.3

Expected Behavior

Same model weights should produce comparable visual understanding regardless of inference server.

[Blaizzy]

Could you share a reproducible example?

[Hassan-A-K]

Here's a minimal reproducible example. You'll need any engineering drawing PDF or detailed document image with small text/stamps.

from mlx_vlm import load, generate
from PIL import Image
import base64, io

model_path = "mlx-community/Qwen3-VL-8B-Instruct-MLX-4bit"
model, processor = load(model_path)

# Load any detailed image (engineering drawing, scanned document, etc.)
# The issue is most visible with images that have small stamps/text annotations
img = Image.open("test_image.png").convert("RGB")
# Resize to ~4 MP to match our test conditions
img = img.resize((2486, 1608))

prompt = """Analyze this image. List all visual features you can detect including:
- Any stamps or seals
- Any checkboxes or form elements  
- Any handwritten annotations or redlines
- Any text overlays

Return a JSON with: {"features_detected": [...], "stamp_detected": true/false}"""

# Using mlx-vlm generate
output = generate(model, processor, prompt, [img], max_tokens=512, temp=0.2)
print("mlx-vlm output:")
print(output)

What we observe:

• mlx-vlm.generate() returns features_detected: [] and stamp_detected: false even when stamps, checkboxes, and redlines are clearly visible
• The same model loaded in LM Studio (which uses mlx-vlm only for vision embedding extraction, then mlx-lm for text generation) correctly identifies all features

Key finding: We verified that prepare_inputs() produces identical pixel_values shape (15600, 1536) — so the image preprocessing is correct. The issue appears to be downstream in the generation pipeline.

We can share a test image privately if needed (engineering drawings are project-specific). Alternatively, any document with a small colored stamp/seal in the corner should reproduce the issue.