Skip to content

Autocleaning emits a spurious safe_int op on unhashable (list/dict/set) columns #878

Description

@paddymul

Summary

The default autoclean config generates a safe_int cleaning op for columns of
unhashable values (lists / dicts / sets).

base_summary_stats short-circuits value_counts to an empty Series for
unhashable columns (buckaroo/customizations/pd_stats_v2.py — restores #843;
value_counts on a list column is meaningless and falls back to an O(n²)
pairwise compare). pd_cleaning_stats then derives its int-parse stats from that
empty value_counts:

coerced_ser = pd.to_numeric(vc.index.values, errors='coerce', ...)   # empty
nan_sum     = (...).sum()                                            # 0
int_parse   = (length - nan_sum) / length                            # 1.0

so int_parse comes out 1.0, clears the 0.3 threshold in
cleaning_gen_ops, and a safe_int op is emitted for a column that is entirely
lists. Casting a list column with safe_int is meaningless.

Reproduction

import pandas as pd
from buckaroo.pluggable_analysis_framework.stat_pipeline import StatPipeline
from buckaroo.customizations.pd_stats_v2 import PD_AUTOCLEAN_DEFAULT_V2

df = pd.DataFrame({'lists': [['a'], ['b'], ['a', 'c'], ['d']]})
sd, errs = StatPipeline(PD_AUTOCLEAN_DEFAULT_V2, unit_test=False).process_df(df)
col = list(sd)[0]
sd[col]['value_counts']   # empty
sd[col]['int_parse']      # 1.0
sd[col]['cleaning_ops']   # [{'symbol': 'safe_int', ...}, {'symbol': 'df'}]

The same value_countsint_parse interaction predates #876 (v1
DefaultSummaryStats also emptied value_counts and PdCleaningStats
computed the same); it became reachable on the default pandas widget once #876
routed it through @stat.

Scope

pd_cleaning_stats × base_summary_stats in pd_stats_v2.py. Gated to the
"default" autoclean config (CleaningConf / PD_AUTOCLEAN_DEFAULT_V2); the
default display path (PD_ANALYSIS_V2) doesn't compute cleaning ops, so this
only fires when a user selects default cleaning. The polars path (pl_cleaning_stats
casts to Int64) returns 0 parses for a list column, so polars likely does not
misfire — worth confirming and aligning the two paths.

Impact

  • safe_int on an unhashable column is a no-op at best, an error at worst, and
    surprising either way.
  • The root fragility is int_parse deriving from value_counts rather than the
    raw series — an empty value_counts should mean "can't decide", not "100% parses".

Found during review of #876; fix is out of scope there.

Metadata

Metadata

Assignees

No one assigned

    Labels

    auto-cleaningauto-cleaning-surpriseThe Auto Cleaning behavior is suprising to users. Keep track of how frequently it surprises users.bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions