[BUG] PMI normalization unusable: requires unconditioned_logprob but API/docs don’t expose a way to provide it #997

Describe the bug

LogLikelihoodAccMetric(normalization=LogProbPMINorm()) appears as a supported normalization option, but it is currently not usable:

• When used directly, the pipeline asserts that unconditioned_logprob must be provided:

  AssertionError: unconditioned_logprob must be provided for PMI normalization
  

• If I attempt to work around this by including both conditioned and unconditioned logprobs in the same list, the evaluation crashes with:

  ValueError: choices_text and choices_logprob must have the same length.
  

There seems to be no documented way to pass unconditioned logprobs per choice, making PMI normalization effectively broken.

To Reproduce

lighteval accelerate \
    "model_name=Qwen/Qwen3-0.6B-Base"\
    "community|readability_ceb_hybrid|0" \
    --custom-tasks lighteval/community_tasks/filipino_evals.py \
    --max-samples 20

OR

A minimal example illustrating the issue:

from lighteval.metrics import LogLikelihoodAccMetric, LogProbPMINorm
from lighteval.pipeline import EvaluationPipeline

metric = LogLikelihoodAccMetric(normalization=LogProbPMINorm())

batch = {
    "choices_text": ["A", "B"],
    # Only conditioned logprobs are possible via current API:
    "choices_logprob": [-1.2, -1.5],
    # There's no supported field to provide unconditioned logprobs here.
}

pipe = EvaluationPipeline(metrics=[metric])
pipe.add_batch(batch)
pipe.evaluate()
# => AssertionError: unconditioned_logprob must be provided for PMI normalization

Attempting a workaround by adding both conditioned and unconditioned values to choices_logprob causes:

ValueError: choices_text and choices_logprob must have the same length.

Expected behavior

One of the following should be supported:

1. Documented input schema:
   Provide a clear way to supply unconditioned logprobs per choice.
   For example:

   "choices_logprob": [-1.2, -1.5],  # conditioned
   "choices_uncond_logprob": [-0.9, -1.1]  # unconditioned

2. Automatic computation:
   If the backend can compute unconditioned logprobs automatically, PMI normalization should trigger that process internally.

3. Better messaging / removal:
   If PMI normalization is not yet implemented, remove LogProbPMINorm() from the normalization options and update docs and error messages accordingly.

Version info

• OS: [Ubuntu 22.04.3 LTS]
• LightEval: [0.11.0]
• Backend: [vllm,accelerate]
• Python: [3.10,3.12]

Additional context / logs

• Error 1:

  AssertionError: unconditioned_logprob must be provided for PMI normalization
  

• Error 2:

  ValueError: choices_text and choices_logprob must have the same length.
  

pmi.log