Why builtin callback doesn't work in xgboost

Question

I tested the code snippet from xgboost documentation and got following error. Does anyone have any idea?

Error message:

KeyError: 'Valid'

The error is due to the fact that early_stop.stopping_history json object doesn't have any 'Valid' key. It seems that the callback hasn't worked properly.

Part of my code can be found bellow.

D_train = xgb.DMatrix(X_train, y_train)
D_valid = xgb.DMatrix(X_valid, y_valid)

# Define a custom evaluation metric used for early stopping.
def eval_error_metric(predt, dtrain: xgb.DMatrix):
    label = dtrain.get_label()
    r = np.zeros(predt.shape)
    gt = predt > 0.5
    r[gt] = 1 - label[gt]
    le = predt <= 0.5
    r[le] = label[le]
    return 'CustomErr', np.sum(r)

early_stopping_rounds = 100

# Specify which dataset and which metric should be used for early stopping.
early_stop = xgb.callback.EarlyStopping(rounds=early_stopping_rounds,
                                        metric_name='CustomErr',
                                        data_name='Train')

clf = xgb.train(
    {'objective': 'binary:logistic',
    'eval_metric': ['error', 'rmse'],
    'tree_method': 'hist'}, D_train,
    evals=[(D_train, 'Train'), (D_valid, 'Valid')],
    feval=eval_error_metric,
    num_boost_round=1000,
    callbacks=[early_stop],
    verbose_eval=False)

dump = clf.get_dump(dump_format='json')
assert len(early_stop.stopping_history['Valid']['CustomErr']) == len(dump)

question from:https://stackoverflow.com/questions/65907783/why-builtin-callback-doesnt-work-in-xgboost

Answer 1

Waitting for answers