What happened + What you expected to happen
ray.data.read_delta silently produces an incorrect schema when reading a Delta table that has Delta Column Mapping enabled. Columns added after column mapping was enabled are stored in the physical parquet files using UUID-based physical names (e.g. col-4d014c4b-7d15-4d7b-a89e-blah-blah). The Delta log contains the mapping from these UUIDs to logical column names, but ray.data.read_delta does not apply this translation. As a result, the returned dataset's schema contains UUID column names for those columns rather than their logical names, and any attempt to select or cast those columns by their logical name raises a KeyError.
Root Cause Chain
ray.data.read_delta calls DeltaTable(path, version=version).file_uris() from deltalake to get parquet paths, then passes them to ray.data.read_parquet.ray.data.read_parquet infers the dataset schema from pq_ds.fragments[0].physical_schema — the first parquet file's raw schema.- For a column-mapping-enabled Delta table, the physical parquet file stores columns using UUID names. The logical name translation lives in the Delta log metadata, which
deltalake 1.6.1 does not apply to the physical schema exposed to Ray.
- The resulting dataset schema contains
col-4d014c4b-7d15-4d7b-a89e-<redacted> double where <string column_name> double is expected.
Observed Schema (excerpt)
is_2_way bool ← last logical name
col-4d014c4b-7d15-4d7b-a89e-40dfdca76a22 double ← should be a logical name
col-8c0bb653-f5db-4b02-a361-d29086e7f2ed double
...
_row-id-col-9e9b0ae3-... int64
_row-commit-version-col-7bc4f86e-... int64
DESCRIBE TABLE confirms real column name exists at ordinal position 356 with type doubleSHOW TBLPROPERTIES confirms delta.columnMapping.mode is enabled on the tableDESCRIBE HISTORY shows no ADD COLUMN operations — the column has existed since table creation; only its physical representation changed when column mapping was enabled- Downgrading
deltalake to 0.x (which applied column mapping translation correctly) resolves the issue
Expected Behavior
ray.data.read_delta should apply the Delta log's column mapping metadata to translate physical UUID column names to their logical names before constructing the dataset schema, consistent with how deltalake 0.x and Databricks' native readers behave.
Further diagnostics
Per Anyscale's suggestion, we supplied a schema on read:
dt = DeltaTable("s3://my-bucket/my-table",
storage_options={"AWS_REGION": "...})
col_map = {
field.metadata.get("delta.columnMapping.physicalName", field.name): field.name
for field in dt.schema().fields
}
ds = ray.data.read_unity_catalog(...)
ds = ds.rename_columns(col_map)This fixes reads (though ideally would not be required), but is incompatible with adding row_filters to read_unity_catalog calls because row filters using logical table names are not mapped to the physical column name. The filter is then silently ignored, causing a full table scan.
Per Anyscale, it seems like we should've hit this error from deltalake, but did not.
Versions / Dependencies
- Ray: 2.55.1
- deltalake: 1.6.1
- PyArrow: 21.0.0
- Python: 3.11
Reproduction script
Nontrivial to reproduce in isolation as this requires a databricks table with column mapping.
Issue Severity
None
What happened + What you expected to happen
ray.data.read_deltasilently produces an incorrect schema when reading a Delta table that has Delta Column Mapping enabled. Columns added after column mapping was enabled are stored in the physical parquet files using UUID-based physical names (e.g.col-4d014c4b-7d15-4d7b-a89e-blah-blah). The Delta log contains the mapping from these UUIDs to logical column names, butray.data.read_deltadoes not apply this translation. As a result, the returned dataset's schema contains UUID column names for those columns rather than their logical names, and any attempt to select or cast those columns by their logical name raises aKeyError.Root Cause Chain
ray.data.read_deltacallsDeltaTable(path, version=version).file_uris()fromdeltalaketo get parquet paths, then passes them toray.data.read_parquet.ray.data.read_parquetinfers the dataset schema frompq_ds.fragments[0].physical_schema— the first parquet file's raw schema.deltalake 1.6.1does not apply to the physical schema exposed to Ray.col-4d014c4b-7d15-4d7b-a89e-<redacted> doublewhere<string column_name> doubleis expected.Observed Schema (excerpt)
DESCRIBE TABLEconfirms real column name exists at ordinal position 356 with typedoubleSHOW TBLPROPERTIESconfirmsdelta.columnMapping.modeis enabled on the tableDESCRIBE HISTORYshows noADD COLUMNoperations — the column has existed since table creation; only its physical representation changed when column mapping was enableddeltalaketo0.x(which applied column mapping translation correctly) resolves the issueExpected Behavior
ray.data.read_deltashould apply the Delta log's column mapping metadata to translate physical UUID column names to their logical names before constructing the dataset schema, consistent with howdeltalake 0.xand Databricks' native readers behave.Further diagnostics
Per Anyscale's suggestion, we supplied a schema on read:
This fixes reads (though ideally would not be required), but is incompatible with adding
row_filters to read_unity_catalog calls because row filters using logical table names are not mapped to the physical column name. The filter is then silently ignored, causing a full table scan.Per Anyscale, it seems like we should've hit this error from deltalake, but did not.
Versions / Dependencies
Reproduction script
Nontrivial to reproduce in isolation as this requires a databricks table with column mapping.
Issue Severity
None