[Regression] TabPFN v2.6 vs. v2.5 has worse log score and other proper scoring rules

[jonaslandsgesell]

Describe the bug

Description

We observed poor performance of TabPFN v2.6 on the ScoringBench in the log score and other proper scoring rules like CRPS, CRLS, CDE and beta energy scores which allows to judge the quality of distributional forecasts - for more discussions see https://arxiv.org/abs/2603.08206

In our benchmarks, v2.6 ranked below v2.5 in log score. One example where the problem occurs is OpenML Dataset 44056 https://www.openml.org/search?type=data&status=active&id=44056

MWE

We unittested our log score implementation against the TabPFN internal implementation and found numerical equivalence between our implementation and criterion.compute_scaled_log_probs(logits) ruling out a log score specific problem on our side.
Here I provide a minimal working example to demonstrate a difference in performance of tabpfnv2.5 to tabpfnv2.6:

Steps/Code to Reproduce

import numpy as np
import openml
import torch
from tabpfn import TabPFNRegressor

# --- SETUP ---
DATASET_ID = 44056
V25, V26 = "v2.5", "v2.6"
MODEL_PATHS = {
    V25: "tabpfn-v2.5-regressor-v2.5_real.ckpt",
    V26: "tabpfn-v2.6-regressor-v2.6_default.ckpt",
}

print(f"Loading dataset {DATASET_ID}...")
dataset = openml.datasets.get_dataset(DATASET_ID)
X, y, _, _ = dataset.get_data(dataset_format="dataframe", target=dataset.default_target_attribute)

# Shuffle before splitting
np.random.seed(42)
perm = np.random.permutation(len(X))
X, y = X.iloc[perm].reset_index(drop=True), np.asarray(y)[perm]

# Keep only numeric columns
numeric_cols = X.select_dtypes(include=[np.number]).columns
X = X[numeric_cols]

# --- DATA QUALITY CHECKS ---
print("\n=== DATA QUALITY CHECKS ===")
print(f"X shape: {X.shape}, y shape: {y.shape}")
print(f"X dtype: {X.dtypes.unique()}, y dtype: {y.dtype}")
print(f"y min: {np.nanmin(y):.6f}, max: {np.nanmax(y):.6f}, mean: {np.nanmean(y):.6f}, std: {np.nanstd(y):.6f}")
print(f"y NaN count: {np.isnan(y).sum()}, Inf count: {np.isinf(y).sum()}")
for col in X.columns:
    x_col = X[col]
    x_arr = x_col.to_numpy(dtype=float)
    nan_count = np.isnan(x_arr).sum()
    print(f"  {col}: min={np.nanmin(x_arr):.6f}, max={np.nanmax(x_arr):.6f}, NaN={nan_count}")


n_train = 2400
X_test_size = 600
X_train, X_test = X[:n_train], X[n_train:n_train + X_test_size]
y_train, y_test = np.asarray(y[:n_train]), np.asarray(y[n_train:n_train + X_test_size])
results_raw = {}

for name, path in MODEL_PATHS.items():
    print(f"\nEvaluating {name}...")
    model = TabPFNRegressor(device="cuda", model_path=path)
    model.fit(X_train, y_train)

    with torch.no_grad():
        out = model.predict(X_test, output_type="full")
        logits, criterion = out["logits"].cuda(), out["criterion"]
        y_t = torch.as_tensor(y_test, device="cuda", dtype=torch.float32).unsqueeze(-1)

        # Compute NLL loss using criterion.forward()
        nll_vals = criterion.forward(logits, y_t).cpu().numpy().flatten()
        
        inf_indices = np.where(np.isinf(nll_vals) | np.isnan(nll_vals))[0]
        
        y_pred = model.predict(X_test)
        residuals = y_test - y_pred

    results_raw[name] = {
        'nll_raw': nll_vals, 
        'residuals': residuals, 'inf_indices': inf_indices
    }

def print_table(title, data_dict, nll_key):
    print(f"\n{title}")
    print("-" * 80)
    print(f"{'Metric':<25} | {V25:<12} | {V26:<12} | Change")
    print("-" * 80)
    metrics = [
        ("Mean NLL",           lambda d: np.mean(d[nll_key])),
        ("Median NLL",         lambda d: np.median(d[nll_key])),
        ("Max NLL",            lambda d: np.max(d[nll_key])),
        ("MAE",                lambda d: np.mean(np.abs(d['residuals']))),
        ("Inf/NaN Count",      lambda d: len(d['inf_indices'])),
    ]
    for label, fn in metrics:
        a, b = fn(data_dict[V25]), fn(data_dict[V26])
        print(f"{label:<25} | {a:<12.4f} | {b:<12.4f} | {b - a:+.4f}")

# --- REPORT 1: ALL SAMPLES ---
print_table("REPORT: ALL SAMPLES", results_raw, 'nll_raw')

# --- REPORT 2: EXCLUDING INFS (UNION MASK) ---
common_mask = np.ones(X_test_size, dtype=bool)
for name in MODEL_PATHS.keys():
    common_mask[results_raw[name]['inf_indices']] = False

results_filtered = {}
for name in MODEL_PATHS.keys():
    results_filtered[name] = {
        k: (v[common_mask] if isinstance(v, np.ndarray) and len(v) == X_test_size else v)
        for k, v in results_raw[name].items()
    }
    results_filtered[name]['inf_indices'] = np.array([]) # Reset for table display

print_table(f"REPORT: EXCLUDING INFS ({np.sum(~common_mask)} samples removed)", results_filtered, 'nll_raw')

Expected Results

Roughly similar NLL values (or better)

Actual Results

python debug_tabpfn_versions.py
Loading dataset 44056...

=== DATA QUALITY CHECKS ===
X shape: (8641, 3), y shape: (8641,)
X dtype: [dtype('float64')], y dtype: uint8
y min: 1.000000, max: 40.000000, mean: 16.863326, std: 12.384738
y NaN count: 0, Inf count: 0
  northing: min=-0.010000, max=3.806000, NaN=0
  easting: min=-0.004000, max=1.560000, NaN=0
  resistivity: min=0.890000, max=166.010000, NaN=0

Evaluating v2.5...

Evaluating v2.6...

REPORT: ALL SAMPLES
--------------------------------------------------------------------------------
Metric                    | v2.5         | v2.6         | Change
--------------------------------------------------------------------------------
Mean NLL                  | -1.0728      | inf          | +inf
Median NLL                | -1.2498      | 12.6195      | +13.8693
Max NLL                   | 4.3556       | inf          | +inf
MAE                       | 1.2312       | 0.8073       | -0.4239
Inf/NaN Count             | 0.0000       | 217.0000     | +217.0000

REPORT: EXCLUDING INFS (217 samples removed)
--------------------------------------------------------------------------------
Metric                    | v2.5         | v2.6         | Change
--------------------------------------------------------------------------------
Mean NLL                  | -1.0275      | 6.6914       | +7.7189
Median NLL                | -1.2132      | 7.8900       | +9.1032
Max NLL                   | 3.5921       | 16.1006      | +12.5085
MAE                       | 1.6193       | 1.0045       | -0.6148
Inf/NaN Count             | 0.0000       | 0.0000       | +0.0000

Versions

PyTorch version: 2.9.1+cu128
CUDA used to build PyTorch: 12.8

Dependency Versions:
--------------------
tabpfn: 7.0.1
torch: 2.9.1
numpy: 2.1.3
scipy: 1.15.3
pandas: 2.3.3
scikit-learn: 1.6.1
typing_extensions: 4.15.0
einops: 0.8.2
huggingface-hub: 0.36.2

[jonaslandsgesell]

I took the diagnostic script from @noahho and adapted it to check some scoring rules via ScoringBench:

"""Diagnostic script to verify whether softmax_temperature and average_before_softmax
affect distributional metrics using ScoringBench.

Uses ScoringBench metrics:
  Point Estimates:  MAE, RMSE, R²
  Distributional:   CRPS, log_score, CRLS, sharpness, dispersion
  Intervals:        coverage_90, interval_score_90, coverage_95, interval_score_95
  Energy Scores:    energy_score_beta_{0.5, 1.0, 1.5, 2.0}
  Calibration:      wcrps_left, wcrps_right, wcrps_center
"""

from __future__ import annotations

import numpy as np
import pandas as pd
import time

from scoringbench.wrappers.tabpfn import TabPFNWrapper
from scoringbench.metrics import compute_metrics


# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------

CONFIGS = [
    {
        "name": "v2.5: softmax_temp=0.9",
        "model_path": "tabpfn-v2.5-regressor-v2.5_real.ckpt",
        "softmax_temperature": 0.9,
        "average_before_softmax": False,
    },
    {
        "name": "v2.5: softmax_temp=1.0",
        "model_path": "tabpfn-v2.5-regressor-v2.5_real.ckpt",
        "softmax_temperature": 1.0,
        "average_before_softmax": False,
    },
    {
        "name": "v2.6: softmax_temp=0.9",
        "model_path": "tabpfn-v2.6-regressor-v2.6_default.ckpt",
        "softmax_temperature": 0.9,
        "average_before_softmax": False,
    },
    {
        "name": "v2.6: softmax_temp=1.0",
        "model_path": "tabpfn-v2.6-regressor-v2.6_default.ckpt",
        "softmax_temperature": 1.0,
        "average_before_softmax": False,
    },
]

# Metrics to report (subset of ScoringBench metrics)
POINT_METRICS = ["mae", "rmse", "r2"]
SCORING_RULES = [
    "crps", "log_score", "crls", "sharpness", "dispersion",
    "coverage_90", "interval_score_90",
    "coverage_95", "interval_score_95",
    "energy_score_beta_0.5", "energy_score_beta_1.0",
    "wcrps_left", "wcrps_right", "wcrps_center",
]

ALL_METRICS = POINT_METRICS + SCORING_RULES


# ---------------------------------------------------------------------------
# Evaluation
# ---------------------------------------------------------------------------

def evaluate_config(X_train, y_train, X_test, y_test, *, config_dict):
    """Fit TabPFN with given config and evaluate on test set using ScoringBench.
    
    Parameters
    ----------
    X_train, y_train : training data
    X_test, y_test : test data
    config_dict : dict with model_path, softmax_temperature, average_before_softmax
    
    Returns
    -------
    dict with metrics from ScoringBench
    """
    # Extract hyperparameters, excluding 'name'
    model_params = {k: v for k, v in config_dict.items() if k != "name"}
    
    # Create and fit model using TabPFNWrapper
    model = TabPFNWrapper(
        n_estimators=8,
        random_state=42,
        **model_params,
    )
    
    t0 = time.time()
    model.fit(X_train, y_train)
    train_time = time.time() - t0
    
    # Get distributional predictions
    y_test_np = np.asarray(y_test, dtype=float)
    dist = model.predict_distribution(X_test)
    
    # Compute all metrics via ScoringBench
    metrics = compute_metrics(dist, y_test_np)
    metrics["train_time"] = train_time
    
    return metrics


# ---------------------------------------------------------------------------
# Results formatting
# ---------------------------------------------------------------------------

def format_metrics_table(results_list):
    """Format results as a nice table.
    
    Parameters
    ----------
    results_list : list of dicts, each with 'config_name' and metric values
    
    Returns
    -------
    str : formatted table
    """
    # Build dataframe
    df = pd.DataFrame(results_list)
    
    # Reorder columns for readability
    col_order = ["config_name", "train_time"]
    for m in POINT_METRICS:
        if m in df.columns:
            col_order.append(m)
    for m in SCORING_RULES:
        if m in df.columns:
            col_order.append(m)
    
    df = df[[c for c in col_order if c in df.columns]]
    
    # Format for display
    output = "\n" + "="*140 + "\n"
    output += "TabPFN Configuration Comparison (ScoringBench)\n"
    output += "="*140 + "\n\n"
    
    # Print with flexible column widths
    for idx, row in df.iterrows():
        output += f"\n{'-'*140}\n"
        output += f"Config {idx+1}: {row['config_name']}\n"
        output += f"{'-'*140}\n"
        
        # Point metrics
        output += "Point Estimates:\n"
        for m in POINT_METRICS:
            if m in row and row[m] is not None:
                output += f"  {m:20s} : {row[m]:>10.6f}\n"
        
        # Scoring rules
        output += "\nScoring Rules (lower generally better, except coverage/dispersion):\n"
        for m in SCORING_RULES:
            if m in row and row[m] is not None:
                output += f"  {m:30s} : {row[m]:>10.6f}\n"
        
        output += f"\n  {'train_time':20s} : {row['train_time']:>10.3f}s\n"
    
    output += "\n" + "="*140 + "\n"
    return output


# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------

def main():
    """Generate synthetic regression data and evaluate TabPFN configs."""
    print("\nGenerating synthetic data...")
    rng = np.random.default_rng(42)
    n_train, n_test, n_features = 100, 200, 2
    
    X = rng.normal(0, 1, (n_train + n_test, n_features))
    w = rng.normal(0, 1, n_features)
    y = X @ w + rng.normal(0, 1, n_train + n_test)
    
    X_train, X_test = X[:n_train], X[n_train:]
    y_train, y_test = y[:n_train], y[n_train:]
    
    print(f"  Train: {len(X_train)} samples × {X_train.shape[1]} features")
    print(f"  Test:  {len(X_test)} samples × {X_test.shape[1]} features\n")
    
    # Evaluate each configuration
    results_list = []
    for i, cfg in enumerate(CONFIGS, 1):
        print(f"Evaluating config {i}/{len(CONFIGS)}: {cfg['name']}...")
        metrics = evaluate_config(X_train, y_train, X_test, y_test, config_dict=cfg)
        
        result_row = {"config_name": cfg["name"]}
        result_row.update(metrics)
        results_list.append(result_row)
    
    # Display results
    table_output = format_metrics_table(results_list)
    print(table_output)
    
    # Summary insights
    print("\nKey Insights:")
    print("-" * 70)
    df = pd.DataFrame(results_list)
    
    # Separate by model version
    v25_rows = df[df["config_name"].str.contains("v2.5")]
    v26_rows = df[df["config_name"].str.contains("v2.6")]
    
    # Model version comparison
    print("\n📊 Model Version Comparison:")
    print("-" * 70)
    
    if not v25_rows.empty and not v26_rows.empty:
        key_metrics = ["crps", "log_score", "crls", "mae", "rmse"]
        
        for metric in key_metrics:
            if metric in df.columns:
                v25_mean = v25_rows[metric].mean()
                v26_mean = v26_rows[metric].mean()
                better_version = "v2.6" if v26_mean < v25_mean else "v2.5"
                improvement = abs(v26_mean - v25_mean) / v25_mean * 100 if v25_mean != 0 else 0
                
                symbol = "✓" if better_version == "v2.6" else "✗"
                print(f"{symbol} {metric:12s}: v2.5={v25_mean:.6f}, v2.6={v26_mean:.6f}  "
                      f"→ {better_version} better ({improvement:+.1f}%)")
    
    # Find best overall configurations
    print("\n" + "-" * 70)
    print("Best Configurations by Metric:")
    print("-" * 70)
    
    # Find best by CRPS
    if "crps" in df.columns:
        best_by_crps = df.loc[df["crps"].idxmin()]
        print(f"✓ Best CRPS:     {best_by_crps['config_name']}")
        print(f"  Value: {best_by_crps['crps']:.6f}")
    
    # Find best by log_score
    if "log_score" in df.columns:
        best_by_logscore = df.loc[df["log_score"].idxmin()]
        print(f"\n✓ Best log_score: {best_by_logscore['config_name']}")
        print(f"  Value: {best_by_logscore['log_score']:.6f}")
    
    # Check calibration (coverage should be near target)
    if "coverage_95" in df.columns:
        print("\n✓ 95% Coverage (target ≈ 0.95):")
        for idx, row in df.iterrows():
            cov = row.get("coverage_95")
            if cov is not None:
                deviation = abs(cov - 0.95)
                print(f"  {row['config_name']:45s}: {cov:.4f} (δ={deviation:.4f})")


if __name__ == "__main__":
    main()

Diagnostic output would be

Evaluating config 1/4: v2.5: softmax_temp=0.9...
Evaluating config 2/4: v2.5: softmax_temp=1.0...
Evaluating config 3/4: v2.6: softmax_temp=0.9...
Evaluating config 4/4: v2.6: softmax_temp=1.0...

============================================================================================================================================
TabPFN Configuration Comparison (ScoringBench)
============================================================================================================================================


--------------------------------------------------------------------------------------------------------------------------------------------
Config 1: v2.5: softmax_temp=0.9
--------------------------------------------------------------------------------------------------------------------------------------------
Point Estimates:
  mae                  :   0.809733
  rmse                 :   1.010033
  r2                   :   0.305924

Scoring Rules (lower generally better, except coverage/dispersion):
  crps                           :   0.572731
  log_score                      :   1.438912
  crls                           :   1.834724
  sharpness                      :   1.051316
  dispersion                     :   0.084058
  coverage_90                    :   0.905000
  interval_score_90              :   4.206678
  coverage_95                    :   0.950000
  interval_score_95              :   4.744653
  energy_score_beta_0.5          :   0.493176
  energy_score_beta_1.0          :   0.572731
  wcrps_left                     :   0.176760
  wcrps_right                    :   0.171457
  wcrps_center                   :   0.112197

  train_time           :      0.372s

--------------------------------------------------------------------------------------------------------------------------------------------
Config 2: v2.5: softmax_temp=1.0
--------------------------------------------------------------------------------------------------------------------------------------------
Point Estimates:
  mae                  :   0.809916
  rmse                 :   1.009952
  r2                   :   0.306035

Scoring Rules (lower generally better, except coverage/dispersion):
  crps                           :   0.573318
  log_score                      :   1.442811
  crls                           :   1.842291
  sharpness                      :   1.121368
  dispersion                     :   0.093319
  coverage_90                    :   0.910000
  interval_score_90              :   4.231658
  coverage_95                    :   0.965000
  interval_score_95              :   4.792137
  energy_score_beta_0.5          :   0.493495
  energy_score_beta_1.0          :   0.573318
  wcrps_left                     :   0.177180
  wcrps_right                    :   0.171502
  wcrps_center                   :   0.112256

  train_time           :      0.144s

--------------------------------------------------------------------------------------------------------------------------------------------
Config 3: v2.6: softmax_temp=0.9
--------------------------------------------------------------------------------------------------------------------------------------------
Point Estimates:
  mae                  :   0.820851
  rmse                 :   1.027976
  r2                   :   0.281045

Scoring Rules (lower generally better, except coverage/dispersion):
  crps                           :   0.585660
  log_score                      :   1.873305
  crls                           :   1.951373
  sharpness                      :   2.901557
  dispersion                     :   0.197051
  coverage_90                    :   0.905000
  interval_score_90              :   4.319762
  coverage_95                    :   0.955000
  interval_score_95              :   4.778510
  energy_score_beta_0.5          :   0.499350
  energy_score_beta_1.0          :   0.585660
  wcrps_left                     :   0.181520
  wcrps_right                    :   0.174616
  wcrps_center                   :   0.114682

  train_time           :      0.189s

--------------------------------------------------------------------------------------------------------------------------------------------
Config 4: v2.6: softmax_temp=1.0
--------------------------------------------------------------------------------------------------------------------------------------------
Point Estimates:
  mae                  :   0.820047
  rmse                 :   1.027226
  r2                   :   0.282094

Scoring Rules (lower generally better, except coverage/dispersion):
  crps                           :   0.586621
  log_score                      :   1.797457
  crls                           :   1.981610
  sharpness                      :   3.202221
  dispersion                     :   0.184006
  coverage_90                    :   0.920000
  interval_score_90              :   4.345280
  coverage_95                    :   0.975000
  interval_score_95              :   4.817917
  energy_score_beta_0.5          :   0.499909
  energy_score_beta_1.0          :   0.586621
  wcrps_left                     :   0.181761
  wcrps_right                    :   0.175072
  wcrps_center                   :   0.114791

  train_time           :      0.191s

============================================================================================================================================


Key Insights:
----------------------------------------------------------------------

📊 Model Version Comparison:
----------------------------------------------------------------------
✗ crps        : v2.5=0.573025, v2.6=0.586140  → v2.5 better (+2.3%)
✗ log_score   : v2.5=1.440861, v2.6=1.835381  → v2.5 better (+27.4%)
✗ crls        : v2.5=1.838508, v2.6=1.966492  → v2.5 better (+7.0%)
✗ mae         : v2.5=0.809825, v2.6=0.820449  → v2.5 better (+1.3%)
✗ rmse        : v2.5=1.009993, v2.6=1.027601  → v2.5 better (+1.7%)

----------------------------------------------------------------------
Best Configurations by Metric:
----------------------------------------------------------------------
✓ Best CRPS:     v2.5: softmax_temp=0.9
  Value: 0.572731

✓ Best log_score: v2.5: softmax_temp=0.9
  Value: 1.438912

✓ 95% Coverage (target ≈ 0.95):
  v2.5: softmax_temp=0.9                       : 0.9500 (δ=0.0000)
  v2.5: softmax_temp=1.0                       : 0.9650 (δ=0.0150)
  v2.6: softmax_temp=0.9                       : 0.9550 (δ=0.0050)
  v2.6: softmax_temp=1.0                       : 0.9750 (δ=0.0250)

It’s been an interesting evolution since the early iterations of ScoringBench in January and initially we sat out to create a benchmark for distributional regression and scoring rules. Since we kept enhancing ScoringBench, it proved useful as a "normalization layer" for different model architectures. With the above example I came to think about supporting the use as integration gate for major releases (which are -in the end- just other models to be benchmarked). Here we see, how one could setup ScoringBench to obtain hard, automated test-cases.

I've adapted the library and the resulting benchmark on openml datasets to be as low-friction as possible.
The goal is to enable quick evaluations and the possibility to 'health check' the output.

[scheidan]

Just cross-referencing this more visual demonstration of the issue: #871 (comment)