Summary
The default autoclean config generates a safe_int cleaning op for columns of
unhashable values (lists / dicts / sets).
base_summary_stats short-circuits value_counts to an empty Series for
unhashable columns (buckaroo/customizations/pd_stats_v2.py — restores #843;
value_counts on a list column is meaningless and falls back to an O(n²)
pairwise compare). pd_cleaning_stats then derives its int-parse stats from that
empty value_counts:
coerced_ser = pd.to_numeric(vc.index.values, errors='coerce', ...) # empty
nan_sum = (...).sum() # 0
int_parse = (length - nan_sum) / length # 1.0
so int_parse comes out 1.0, clears the 0.3 threshold in
cleaning_gen_ops, and a safe_int op is emitted for a column that is entirely
lists. Casting a list column with safe_int is meaningless.
Reproduction
import pandas as pd
from buckaroo.pluggable_analysis_framework.stat_pipeline import StatPipeline
from buckaroo.customizations.pd_stats_v2 import PD_AUTOCLEAN_DEFAULT_V2
df = pd.DataFrame({'lists': [['a'], ['b'], ['a', 'c'], ['d']]})
sd, errs = StatPipeline(PD_AUTOCLEAN_DEFAULT_V2, unit_test=False).process_df(df)
col = list(sd)[0]
sd[col]['value_counts'] # empty
sd[col]['int_parse'] # 1.0
sd[col]['cleaning_ops'] # [{'symbol': 'safe_int', ...}, {'symbol': 'df'}]
The same value_counts → int_parse interaction predates #876 (v1
DefaultSummaryStats also emptied value_counts and PdCleaningStats
computed the same); it became reachable on the default pandas widget once #876
routed it through @stat.
Scope
pd_cleaning_stats × base_summary_stats in pd_stats_v2.py. Gated to the
"default" autoclean config (CleaningConf / PD_AUTOCLEAN_DEFAULT_V2); the
default display path (PD_ANALYSIS_V2) doesn't compute cleaning ops, so this
only fires when a user selects default cleaning. The polars path (pl_cleaning_stats
casts to Int64) returns 0 parses for a list column, so polars likely does not
misfire — worth confirming and aligning the two paths.
Impact
safe_int on an unhashable column is a no-op at best, an error at worst, and
surprising either way.
- The root fragility is
int_parse deriving from value_counts rather than the
raw series — an empty value_counts should mean "can't decide", not "100% parses".
Found during review of #876; fix is out of scope there.
Summary
The default autoclean config generates a
safe_intcleaning op for columns ofunhashable values (lists / dicts / sets).
base_summary_statsshort-circuitsvalue_countsto an empty Series forunhashable columns (
buckaroo/customizations/pd_stats_v2.py— restores #843;value_countson a list column is meaningless and falls back to an O(n²)pairwise compare).
pd_cleaning_statsthen derives its int-parse stats from thatempty
value_counts:so
int_parsecomes out1.0, clears the0.3threshold incleaning_gen_ops, and asafe_intop is emitted for a column that is entirelylists. Casting a list column with
safe_intis meaningless.Reproduction
The same
value_counts→int_parseinteraction predates #876 (v1DefaultSummaryStatsalso emptiedvalue_countsandPdCleaningStatscomputed the same); it became reachable on the default pandas widget once #876
routed it through
@stat.Scope
pd_cleaning_stats×base_summary_statsinpd_stats_v2.py. Gated to the"default" autoclean config (
CleaningConf/PD_AUTOCLEAN_DEFAULT_V2); thedefault display path (
PD_ANALYSIS_V2) doesn't compute cleaning ops, so thisonly fires when a user selects default cleaning. The polars path (
pl_cleaning_statscasts to
Int64) returns 0 parses for a list column, so polars likely does notmisfire — worth confirming and aligning the two paths.
Impact
safe_inton an unhashable column is a no-op at best, an error at worst, andsurprising either way.
int_parsederiving fromvalue_countsrather than theraw series — an empty
value_countsshould mean "can't decide", not "100% parses".Found during review of #876; fix is out of scope there.