作者:??塵沙櫻落

樹模型初始化技巧

大家都知道神經網絡訓練的提升分數的技巧之一就是：

• 依據驗證集合的效果，來調整learning rate的大小，從而獲得更好的效果;

但我們在訓練樹模型的時候卻往往是按下面的方式操作：

• 找到一個還不錯的learning rate然后訓練，不行就自動換一個learning rate；

那么能不能把神經網絡的這個用于LightGBM呢？答案是：

可以！！！！

而且效果還有一定提升哦！那么怎么做呢？

傳統策略

找一個線下調試不錯的學習率，訓完結束！

import?pandas?as?pd import?numpy?as?np import?lightgbm?as?lgbm from?sklearn.model_selection?import?KFold from?sklearn.metrics?import?mean_squared_error #####?1.讀取數據 train?=?pd.read_csv("...") test?=?pd.read_csv("...") #####?2.N折訓練測試 cont_features?=?[col?for?col?in?train.columns?if?col.startswith("cont")] len(cont_features)? y?=?train["target"] kf?=?KFold(n_splits=5,?shuffle=True,?random_state=1) oof?=?np.zeros(len(train)) score_list?=?[] fold?=?1 test_preds?=?[]for?train_index,?test_index?in?kf.split(train):X_train,?X_val?=?train.iloc[train_index],?train.iloc[test_index]y_train,?y_val?=?y.iloc[train_index],?y.iloc[test_index]?y_pred_list?=?[]for?seed?in?[1]:dtrain?=?lgbm.Dataset(X_train[cont_features],?y_train)dvalid?=?lgbm.Dataset(X_val[cont_features],?y_val)print(seed)params?=?{"objective":?"regression","metric":?"rmse","verbosity":?-1,"boosting_type":?"gbdt","feature_fraction":0.5,"num_leaves":?200,"lambda_l1":2,"lambda_l2":2,"learning_rate":0.01,'min_child_samples':?50,"bagging_fraction":0.7,"bagging_freq":1}params["seed"]?=?seedmodel?=?lgbm.train(params,dtrain,valid_sets=[dtrain,?dvalid],verbose_eval=100,num_boost_round=100000,early_stopping_rounds=100)y_pred_list.append(model.predict(X_val[cont_features]))test_preds.append(model.predict(test[cont_features]))?oof[test_index]?=?np.mean(y_pred_list,axis=0)????score?=?np.sqrt(mean_squared_error(y_val,?oof[test_index]))score_list.append(score)print(f"RMSE?Fold-{fold}?:?{score}")fold+=1np.mean(score_list)

Tricks版本

大致思路：

• 用一個較大的learning rate學習得到初始版本模型1；

• 用一個較小的learning rate在模型1上繼續訓練得到模型2；

• ...

import?pandas?as?pd import?numpy?as?np import?lightgbm?as?lgbm from?sklearn.model_selection?import?KFold from?sklearn.metrics?import?mean_squared_error #####?1.讀取數據 train?=?pd.read_csv("...") test?=?pd.read_csv("...") #####?2.N折訓練測試 cont_features?=?[col?for?col?in?train.columns?if?col.startswith("cont")] len(cont_features)? y?=?train["target"] kf?=?KFold(n_splits=5,?shuffle=True,?random_state=1) oof?=?np.zeros(len(train)) score_list?=?[] fold?=?1 test_preds?=?[]for?train_index,?test_index?in?kf.split(train):X_train,?X_val?=?train.iloc[train_index],?train.iloc[test_index]y_train,?y_val?=?y.iloc[train_index],?y.iloc[test_index]X_train?=?X_train.abs()y_pred_list?=?[]for?seed?in?[1]:dtrain?=?lgbm.Dataset(X_train[cont_features],?y_train)dvalid?=?lgbm.Dataset(X_val[cont_features],?y_val)print(seed)params?=?{"objective":?"regression","metric":?"rmse","verbosity":?-1,"boosting_type":?"gbdt","feature_fraction":0.5,"num_leaves":?200,"lambda_l1":2,"lambda_l2":2,"learning_rate":0.01,'min_child_samples':?50,"bagging_fraction":0.7,"bagging_freq":1}params["seed"]?=?seedmodel?=?lgbm.train(params,dtrain,valid_sets=[dtrain,?dvalid],verbose_eval=100,num_boost_round=100000,early_stopping_rounds=100)#####?3.?額外的策略dtrain?=?lgbm.Dataset(X_train[cont_features],?y_train)dvalid?=?lgbm.Dataset(X_val[cont_features],?y_val)params?=?{"objective":?"regression","metric":?"rmse","verbosity":?-1,"boosting_type":?"gbdt","feature_fraction":0.5,"num_leaves":?300,"lambda_l1":2,"lambda_l2":2,"learning_rate":0.003,'min_child_samples':?50,"bagging_fraction":0.7,"bagging_freq":1}params["seed"]?=?seedmodel?=?lgbm.train(params,dtrain,valid_sets=[dtrain,?dvalid],verbose_eval=100,num_boost_round=1000,early_stopping_rounds=100,init_model?=?model)y_pred_list.append(model.predict(X_val[cont_features]))test_preds.append(model.predict(test[cont_features]))oof[test_index]?=?np.mean(y_pred_list,axis=0)????score?=?np.sqrt(mean_squared_error(y_val,?oof[test_index]))score_list.append(score)print(f"RMSE?Fold-{fold}?:?{score}")fold+=1np.mean(score_list)

小結

通過上面的方案,作者在數據實踐中都獲得了比傳統策略要更好的線下和線上效果！妙哉！

參考文獻

https://www.kaggle.com/fatihozturk/lgbm-model-initialisation-trick

?

往期精彩回顧適合初學者入門人工智能的路線及資料下載機器學習及深度學習筆記等資料打印機器學習在線手冊深度學習筆記專輯《統計學習方法》的代碼復現專輯 AI基礎下載機器學習的數學基礎專輯 本站知識星球“黃博的機器學習圈子”（92416895） 本站qq群704220115。 加入微信群請掃碼：

總結

以上是生活随笔為你收集整理的【数据竞赛】Kaggle GM秘技：树模型初始化技巧的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。