Optional[T] generates Python None instead of JSON null inside DSL containers

[ErenAta16]

Describe the issue as clearly as possible

Optional[T] and Union[T, None] inside a DSL container generate the Python spelling None rather than the JSON spelling null, so the resulting regex accepts documents json.loads rejects and rejects the ones it accepts.

This is the same class of bug as #1942 (booleans generating True/False instead of true/false) and #1960 (date/time not being quoted), and it is the last of the three JSON literals still using Python syntax.

Steps/code to reproduce the bug

import json, re
from typing import List, Optional
from outlines.types.dsl import python_types_to_terms, to_regex

rx = to_regex(python_types_to_terms(List[Optional[int]]))
print(rx)

for s in ["[1]", "[None]", "[null]", "[1, None]", "[1, null]"]:
    matches = re.fullmatch(rx, s) is not None
    try:
        json.loads(s)
        valid_json = True
    except Exception:
        valid_json = False
    print(f"{s:12} matches={matches!s:5} valid_json={valid_json}")

Expected result

The generated pattern accepts [null] and rejects [None], matching what a JSON parser accepts.

Error message

No exception. The generation is silently wrong:

\[(([+-]?(0|[1-9][0-9]*))|(None))(,\ (([+-]?(0|[1-9][0-9]*))|(None)))*\]

[1]          matches=True  valid_json=True
[None]       matches=True  valid_json=False
[null]       matches=False valid_json=True
[1, None]    matches=True  valid_json=False
[1, null]    matches=False valid_json=True

Every row where the two columns disagree is a case where constrained generation is steering the model toward output that cannot be parsed. [null], the only correct spelling, is the one the pattern forbids.

The same holds for the other containers and for the explicit union spelling:

List[Optional[str]]        ["a"]=ok    [null]=rejected   [None]=accepted
List[Optional[bool]]       [True]=ok   [null]=rejected   [None]=accepted
Dict[str, Optional[int]]   {"k":1}=ok  {"k":null}=rejected  {"k":None}=accepted
List[Union[int, None]]     [1]=ok      [null]=rejected   [None]=accepted

Outlines/Python version information

Reproduced on main at be2cd151855c64a81262c4daace2428400b109ff, Python 3.12, Windows.

Context for solving the bug

The term comes from _handle_union in src/outlines/types/dsl.py:

if has_none:
    # `None` is a keyword, not a string value: keep it a `Regex` (like
    # `True`/`False`) so it is not JSON-quoted when the union ends up
    # nested inside a container type.
    terms.append(Regex("None"))

The reasoning in that comment is half right. Keeping it a Regex correctly avoids "None" coming out quoted as a string, which would be a different wrong answer. What it misses is that inside a JSON container the spelling itself has to change, and the precedent it cites, True/False, is exactly the spelling #1942 identified as a bug.

Worth noting the standalone case is a separate question and I would leave it alone. to_regex(python_types_to_terms(Optional[int])) producing None outside any container is consistent with bool producing True, and #1947 and #1961 both settled on changing the spelling only in container context while leaving the standalone term as it is. The same split applies here, so whatever shape the boolean fix ends up taking should be able to cover None without a second mechanism.

I am happy to open a PR once the boolean one lands, since both need to hook the same place and doing them separately would mean writing the container-context check twice.