Variables not in any trees have positive gain/cover/weight

[jhaneyrf]

I was converting an XGBoost classifier model to PMML, and I noticed that about 40 of the columns in the model.feature_names list were not present in the PMML file.

I did a little more research. I usually load my model as a Booster() object, but when I loaded it as XGBClassifier() and looked at the feature_importances_ attribute, I saw that those 40 variables all had zero importance.

So I went back to loading from a Booster() and output get_dump() to a text file. I could not find any reference to those 40 columns in that file.

So then I ran the get_score method on all five importance types. For each importance type, these unused variables had a positive score.

Is this a known issue? Is there some place that documents why this behavior should be expected?

It appears that my PMML file is correct even though it excludes those 40 variables. I'm just wondering why those 40 variables wound up in the feature_names or feature_names_in_ list for this model if they are not used in any of the final trees.

I did all of this in XGBoost 2.1.3.

Thanks

[trivialfis]

So then I ran the get_score method on all five importance types. For each importance type, these unused variables had a positive score.

This should not happen. Could you please share something I can use to reproduce?

I tested with this small example, did not observe the issue:

import xgboost as xgb
from sklearn.datasets import make_classification

n_features = 16

X, y = make_classification(
    n_samples=1024,
    n_features=n_features,
    random_state=1,
    n_classes=4,
    n_redundant=0,
    n_informative=n_features,
)

clf = xgb.XGBClassifier(max_depth=2, n_estimators=2)
clf.fit(X, y)
clf.save_model("clf.json")
print(clf.feature_importances_)

booster = xgb.Booster(model_file="clf.json")
for tree in booster.get_dump():
    print(tree)

scores = booster.get_score()
print(scores)

[jhaneyrf]

Since I can't share the model that I saw the issue in, it might take me some time to create a model that reproduces this issue. But I will try.

[trivialfis]

Since I can't share the model that I saw the issue in

Feel free to change the feature names and values to something random by editing the json files. Python has builtin json module. Since the issue is having positive feature importance for unused features, changing the name (consistently) and tree split values should not affect the issue. Unused features continue to be unused regardless of the names.