Uncertainty Toolbox: a Python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization

A Library for Uncertainty Quantification.

A collection of research and application papers of (uncertainty) calibration techniques.

A toolkit for visualizations in materials informatics.

Calibration library and code for the paper: Verified Uncertainty Calibration. Ananya Kumar, Percy Liang, Tengyu Ma. NeurIPS 2019 (Spotlight).

(ECCV 2022) BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networks

Code for evaluating uncertainty estimation methods for Transformer-based architectures in natural language understanding tasks.

Source code for our paper: "LoGU: Long-form Generation with Uncertainty Expressions".

A project to train your model from scratch or fine-tune a pretrained model using the losses provided in this library to improve out-of-distribution detection and uncertainty estimation performances. Calibrate your model to produce enhanced uncertainty estimations. Detect out-of-distribution data using the defined score type and threshold.

Scoring rules like the Brier Score (Mean Squared Error, Quadratic Score) and Log Loss (Cross-Entropy, Negative Log-Likelihood, Logarithmic Score) can favor incorrect predictions. To address this limitation, the Probabilistic Brier Score (PBS) and Probabilistic Logarithmic Loss (PLL) have been introduced for probabilistic classifiers.

Service to examine data processing pipelines (e.g., machine learning or deep learning pipelines) for uncertainty consistency (calibration), fairness, and other safety-relevant aspects.

Calibration of Few-Shot Classification Tasks: Mitigating Misconfidence from Distribution Mismatch, IEEE Access vol.10

Truth Discovery Promotes Uncertainty Calibration of DNNs (UAI 2021)

A modular Python workflow for assimilating observed land-subsidence data into an ensemble of geomechanical simulations with the Ensemble Smoother with Multiple Data Assimilation (ES-MDA).

MSc thesis: evaluating & improving uncertainty calibration in retrieval-augmented LLMs. Compares 5 embedding models (BGE beats OpenAI) and cuts ECE ~32% via isotonic regression on real financial filings.

GranularVAR: Multi-scale video understanding with augmentation-graded contrastive learning and calibrated uncertainty. V-JEPA 2 encoder + granularity-conditioned decoder. GWU MS Data Science Capstone, Spring 2026.

Risk-based sequential stopping: when to invoke an LLM in streaming systems. Code for ECML PKDD 2026 (Research Track). DOI: 10.5281/zenodo.20783298

Reproducible MEDAI deferral simulation (AIRI 2026). Synthetic research code.

[MSc Thesis, UPC ETSETB 2026] Uncertainty-Guided LoRA Adaptation of Vision Transformers. Calibration via probability-space consistency (PACE-KL).

Multimodal BBQ (bias) visual question answering: pick the right answer from image + context + question + 3 options, and answer "unknown" when evidence is insufficient. Offline, open-weights Qwen2.5-VL (4-bit) + balanced chain-of-thought. Public Balanced Accuracy 0.883.