:[gi3' dZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZmZmZmZmZmZmZddlZddlmZddlmZmZmZmZmZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&dd l'm(Z(dd l)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1dd l2m3Z3m4Z4m5Z5m6Z6ddl7m8Z8GddZ9dee:deee!ddfdZ;dee<de=fdZ>GddeZ?e eegeejejffZAeeeeed?ee:d@ee:dAee<dBeee.innerws**, ""$    !+*     !}d+CvuMB B , %&%  r4)r parametersnpndarrayrr )rKr[rZrYs` @r5_objective_decoratorr^UsU>4++J!Z/K!RZZ!'!eBJJ .innersY""$##% ;;! ==$vw"77 7}}d67&IIIr4r\r]rr strfloat)rKrZs` r5_metric_decoratorrfs5JrzzJGJc5j8IJ Lr4n_jobscjdtjdtdttt fffd }|S)z#Decorate a learning to rank metric.r`rNr8c |j |jd jdkr tdt j jdz }g}|j }|jdk(}dtdtf fd }ntj}t| 5}td jD]%} |j|| } |j| 't|D]\} } | j!|| < ddd|r j"|j%fS j"t j&|| fS#1swYJxYw) N group_ptrz;Invalid `group_ptr`. Likely caused by invalid qid or group.rrir8cd|dz }|}||}||}|jdk(ry||S)Nr?)rS) rlbeginendgygprKrjr`rDs r5taskz1ltr_metric_decorator..inner..tasksNa!e$EA,Cc"Bs#Bww!|B< r4) max_workers)weights)rR get_uint_inforSrTr\empty get_groupintreos cpu_countrrangesubmitappend enumerateresultr.meanaverage)r`rNscoresfuturesrb no_weightrsworkersexecutorrlfrjrDrKrgs` @@r5rZz#ltr_metric_decorator..innersD""$))+6 >>A M )..1,-""$KK1$  C E #.&BLLN G 41inn-OOD!,q!."'*1HHJq + 5 ==&++-/ /}}bjj@@@5 4s ` n_jobs : a0 Number of parallel threads used to run xgboost. When used with other Scikit-Learn algorithms like grid search, you may choose which algorithm to parallelize and balance the threads. Creating thread contention will significantly slow down both algorithms. gamma : z (min_split_loss) Minimum loss reduction required to make a further partition on a leaf node of the tree. min_child_weight : z[ Minimum sum of instance weight(hessian) needed in a child. max_delta_step : z\ Maximum delta step we allow each tree's weight estimation to be. subsample : zK Subsample ratio of the training instance. sampling_method : aI Sampling method. Used only by the GPU version of ``hist`` tree method. - ``uniform``: Select random training instances uniformly. - ``gradient_based``: Select random training instances with higher probability when the gradient and hessian are larger. (cf. CatBoost) colsample_bytree : z[ Subsample ratio of columns when constructing each tree. colsample_bylevel : zM Subsample ratio of columns for each level. colsample_bynode : zF Subsample ratio of columns for each split. reg_alpha : zM L1 regularization term on weights (xgb's alpha). reg_lambda : zT L2 regularization term on weights (xgb's lambda). scale_pos_weight : zG Balancing of positive and negative weights. base_score : zZ The initial prediction score of all instances, global bias. random_state : aE Random number seed. .. note:: Using gblinear booster with shotgun updater is nondeterministic as it uses Hogwild algorithm. missing : float Value in the data which needs to be present as a missing value. Default to :py:data:`numpy.nan`. num_parallel_tree: zG Used for boosting random forest. monotone_constraints : z Constraint of variable monotonicity. See :doc:`tutorial ` for more information. interaction_constraints : a Constraints for interaction representing permitted interactions. The constraints must be specified in the form of a nested list, e.g. ``[[0, 1], [2, 3, 4]]``, where each inner list is a group of indices of features that are allowed to interact with each other. See :doc:`tutorial ` for more information importance_type: a9 The feature importance type for the feature_importances\_ property: * For tree model, it's either "gain", "weight", "cover", "total_gain" or "total_cover". * For linear model, only "weight" is defined and it's the normalized coefficients without bias. device : z .. versionadded:: 2.0.0 Device ordinal, available options are `cpu`, `cuda`, and `gpu`. validate_parameters : z Give warnings for unknown parameter. enable_categorical : bool See the same parameter of :py:class:`DMatrix` for details. feature_types : z .. versionadded:: 1.7.0 Used for specifying feature types without constructing a dataframe. See :py:class:`DMatrix` for details. max_cat_to_onehot : aG .. versionadded:: 1.6.0 .. note:: This parameter is experimental A threshold for deciding whether XGBoost should use one-hot encoding based split for categorical data. When number of categories is lesser than the threshold then one-hot encoding is chosen, otherwise the categories will be partitioned into children nodes. Also, `enable_categorical` needs to be set to have categorical feature support. See :doc:`Categorical Data ` and :ref:`cat-param` for details. max_cat_threshold : a .. versionadded:: 1.7.0 .. note:: This parameter is experimental Maximum number of categories considered for each split. Used only by partition-based splits for preventing over-fitting. Also, `enable_categorical` needs to be set to have categorical feature support. See :doc:`Categorical Data ` and :ref:`cat-param` for details. multi_strategy : a .. versionadded:: 2.0.0 .. note:: This parameter is working-in-progress. The strategy used for training multi-target models, including multi-target regression and multi-class classification. See :doc:`/tutorials/multioutput` for more information. - ``one_output_per_tree``: One model for each target. - ``multi_output_tree``: Use multi-target trees. eval_metric : al .. versionadded:: 1.6.0 Metric used for monitoring the training result and early stopping. It can be a string or list of strings as names of predefined metric in XGBoost (See doc/parameter.rst), one of the metrics in :py:mod:`sklearn.metrics`, or any other user defined metric that looks like `sklearn.metrics`. If custom objective is also provided, then custom metric should implement the corresponding reverse link function. Unlike the `scoring` parameter commonly used in scikit-learn, when a callable object is provided, it's assumed to be a cost function and by default XGBoost will minimize the result during early stopping. For advanced usage on Early stopping like directly choosing to maximize instead of minimize, see :py:obj:`xgboost.callback.EarlyStopping`. See :doc:`/tutorials/custom_metric_obj` and :ref:`custom-obj-metric` for more information. .. code-block:: python from sklearn.datasets import load_diabetes from sklearn.metrics import mean_absolute_error X, y = load_diabetes(return_X_y=True) reg = xgb.XGBRegressor( tree_method="hist", eval_metric=mean_absolute_error, ) reg.fit(X, y, eval_set=[(X, y)]) early_stopping_rounds : a .. versionadded:: 1.6.0 - Activates early stopping. Validation metric needs to improve at least once in every **early_stopping_rounds** round(s) to continue training. Requires at least one item in **eval_set** in :py:meth:`fit`. - If early stopping occurs, the model will have two additional attributes: :py:attr:`best_score` and :py:attr:`best_iteration`. These are used by the :py:meth:`predict` and :py:meth:`apply` methods to determine the optimal number of trees during inference. If users want to access the full model (including trees built after early stopping), they can specify the `iteration_range` in these inference methods. In addition, other utilities like model plotting can also use the entire model. - If you prefer to discard the trees after `best_iteration`, consider using the callback function :py:class:`xgboost.callback.EarlyStopping`. - If there's more than one item in **eval_set**, the last entry will be used for early stopping. If there's more than one metric in **eval_metric**, the last metric will be used for early stopping. callbacks : a List of callback functions that are applied at end of each iteration. It is possible to use predefined callbacks by using :ref:`Callback API `. .. note:: States in callback are not preserved during training, which means callback objects can not be reused for multiple training sessions without reinitialization or deepcopy. .. code-block:: python for params in parameters_grid: # be sure to (re)initialize the callbacks before each run callbacks = [xgb.callback.LearningRateScheduler(custom_rates)] reg = xgboost.XGBRegressor(**params, callbacks=callbacks) reg.fit(X, y) kwargs : a Keyword arguments for XGBoost Booster object. Full documentation of parameters can be found :doc:`here `. Attempting to set a parameter via the constructor args and \*\*kwargs dict simultaneously will result in a TypeError. .. note:: \*\*kwargs unsupported by scikit-learn \*\*kwargs is unsupported by scikit-learn. We do not guarantee that parameters passed via this argument will interact properly with scikit-learn. a- .. note:: Custom objective function A custom objective function can be provided for the ``objective`` parameter. In this case, it should have the signature ``objective(y_true, y_pred) -> [grad, hess]`` or ``objective(y_true, y_pred, *, sample_weight) -> [grad, hess]``: y_true: array_like of shape [n_samples] The target values y_pred: array_like of shape [n_samples] The predicted values sample_weight : Optional sample weights. grad: array_like of shape [n_samples] The value of the gradient for each sample point. hess: array_like of shape [n_samples] The value of the second derivative for each sample point headeritemsextra_parametersend_notec\dtdtfddtdtffd }|S)aObtain documentation for Scikit-Learn wrappers Parameters ---------- header: str An introducion to the class. items : list A list of common doc items. Available items are: - estimators: the meaning of n_estimators - model: All the other parameters - objective: note for customized objective extra_parameters: str Document for class specific parameters, placed at the head. end_note: str Extra notes put to the end.itemr8c0tttd}||S)zReturn selected item estimatorsmodel objective)__estimator_doc __model_doc__custom_obj_note)r__docs r5get_docz"xgboost_model_doc..get_doc!s * *  T{r4clscdg}r|j|jDcgc] }| c}r|jdzg}|j|dj||_|Scc}w)Nz Parameters ---------- z< See :doc:`/python/sklearn_estimator` for more information. r)r~extendjoinr1) rdocrlfull_docrrrrrs r5adddocz!xgboost_model_doc..adddoc*s    JJ' ( .1GAJ./  JJx U U  ggh'  /sA?)rdr )rrrrrrs```` @r5xgboost_model_docr s5,ccDT& Mr4missingXygroupqidrF base_marginfeature_weightseval_setsample_weight_eval_setbase_margin_eval_set eval_groupeval_qidcreate_dmatrixenable_categorical feature_typesc | ||||||||||d }|dn t|dttdtdtffd }||| d} || d } || d } || d } g}t |D]r\}\}}t ||u||u| ||u| ||u| ||u| ||ufr|j |D| ||| || || || ||||| }|j |tt|}t|Dcgc]}d | }}tt||}||fStd| | | | fDr tdg}||fScc}w)zXConvert array_like evaluation matrices into DMatrix. Perform validation on the way.N) datalabelrrrbrrrrrrefrmetanamer8c t|dgzSt|k7r t|dddt|z|S)Nz.'s length does not equal `eval_set`'s length, z expecting , got )lenrT)rr n_validations r5validate_or_nonez3_wrap_evaluation_matrices..validate_or_nonedsY <6L( ( t9 $&FG|nF3t9+>?  r4rrrr) rrrbrrrrrrr validation_c3$K|]}|du ywrHr3).0rs r5 z,_wrap_evaluation_matrices..s!   szL`eval_set` is not set but one of the other evaluation meta info is not None.) rr r rdrallr~r|listzipanyrT)rrrrrrFrrrrrrrrrr train_dmatrixrevalsrlvalid_Xvalid_ymnevals eval_namesrs @r5_wrap_evaluation_matricesr@s(#  '-#  M!(1c(mLx1!1 "$<"  0 "8 &j,? #Hj9%.x%8 !A!qLqL*1->(+{:qMU*QK3&   ]+" !1!4$Q-  4Q 7#'9"/%  Q5&96U16v?A A3' ?S +," %   '$     % %@s Edz3Implementation of the Scikit-Learn API for XGBoost.rcSeZdZddddddddddddddddddddddddejddddddddddddddf'deedeedeedeedeedeed eed e d eed eed eedeedeedeedeedeedeedeedeedeedeedeedeedee ejjejjefdedeedee eeefefdee eeeefdeed eed!eed"ed#eed$eed%eed&eed'ee eeeefd(eed)eeed*ed+dfRd,Zd+eeeffd-Zd+efd.Zd+efd/Zd0ed+dfd1Zd]d3ed+eeefffd4 Zd+eeeffd5Zd+efd6Zd+efd7Zd8e ee jBfd+dfd9Z"ejDjFe"_#d8e$d+dfd:Z%ejJjFe%_#d;e&d+dfd<Z'd ee edefd0eeefd+e(ee eedfee)eeefffd=Z*d>ee+d*ed+e+fd?Z,d@ejZd+dfdAZ.e/dddd2dddddBdCe0dDe0dEee0dFee0dGeee(e0e0fdHee eefdIee eedfdJeee0dKeee0dLee0d+dfdMZ1d+efdNZ2dOee3d+e3fdPZ4 d^dCe0dQedRedFee0dOee3d+e0f dSZ5 d_dCe0dOee3d+ejlfdTZ7d+eeeeeefffdUZ8e9d+efdVZ:e9d+ejlfdWZ;e9d+efdXZe9d+ejlfd[Z?e9d+ejlfd\Z@xZAS)`XGBModelNF max_depth max_leavesmax_bin grow_policy learning_rate n_estimators verbosityrboosterr<rggammamin_child_weightmax_delta_step subsamplesampling_methodcolsample_bytreecolsample_bylevelcolsample_bynode reg_alpha reg_lambdascale_pos_weight base_score random_staternum_parallel_treemonotone_constraintsinteraction_constraintsimportance_typedevicevalidate_parametersrrmax_cat_to_onehotmax_cat_thresholdmulti_strategy eval_metricr6r7kwargsr8c( \ts td||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ ||_||_||_||_||_||_||_||_||_||_||_||_||_| |_||_||_||_||_||_ | |_!|!|_"|"|_#|#|_$|$|_%|%|_&|&|_'|'|_(|(r|(|_)yy)Nz9sklearn needs to be installed in order to use this module)*r ImportErrorrrrrrrrrrr<rrrrrrrrrrrrrrrrgrrrrrrrrrrrr6r7r))rIrrrrrrrrrr<rgrrrrrrrrrrrrrrrrrrrrrrrrrrr6r7rs) r5__init__zXGBModel.__init__sRZ!K )""$ &*" & 0,". 0!2 0"$ 0$ !2( $8!'>$. #6 "4*!2!2,&%:""  DK r4cnddd}t|dr#|jjddk(rd|d<|S)z+Tags used for scikit-learn data validation.T) allow_nan no_validationrupdatershotgunnon_deterministic)hasattrrget)rItagss r5 _more_tagszXGBModel._more_tagss:!D9 4 "t{{y'AY'N(,D$ % r4ct|dS)N_Booster)rrIs r5__sklearn_is_fitted__zXGBModel.__sklearn_is_fitted__stZ((r4cV|jsddlm}|d|jS)zGet the underlying xgboost Booster of this model. This will raise an exception when fit was not called Returns ------- booster : a xgboost booster of underlying model r)NotFittedErrorz)need to call fit or load_model beforehand)rsklearn.exceptionsrr)rIrs r5 get_boosterzXGBModel.get_boosters)))+ 9 !LM M}}r4paramsc 4|s|S|jD]A\}}t||rt||| t|dsi|_||j|<C|j r/|j }|j j||S)a*Set the parameters of this estimator. Modification of the sklearn method to allow unknown kwargs. This allows using the full range of xgboost parameters that are not defined as member variables in sklearn grid search. Returns ------- self r)rrsetattrrrget_xgb_paramsr set_param)rIrkeyvaluer[s r5 set_paramszXGBModel.set_params(sK!,,.JCtS!c5)tX."$DK#( C )  % % ',,.J     ( ( 4 r4Tdeepct||}tj|}|jjd|_|j |jj||t |dr5t|jtr|j |jt|dtjjrD|djtjtjj |d<|St|dtjj"rKt%|dj'tjtjj |d<|S)zGet parameters.rrr)super get_paramscopy __class__ __bases__updater isinstancerdictr\random RandomStaterandintiinfoint32max Generatorryintegers)rIr rcprs r5rzXGBModel.get_paramsGs"#D) YYt_||--a0  bll--b$78 4 "z$++t'D MM$++ & f^,bii.C.C D%+N%;%C%C"&&&F> " ~. 0C0C D%(~&//0B0F0FG&F> " r4c|j}hd}i}|jD]\}}||vs t|r|||<|S)z Get xgboost specific parameters.>rrr7rrrrr6)rrcallable)rIrwrapper_specificfilteredkvs r5rzXGBModel.get_xgb_paramsbsP!%!2  LLNDAq((! #r4c>|jtS|jS)z+Gets the number of xgboost boosting rounds.)rDEFAULT_N_ESTIMATORSrs r5get_num_boosting_roundsz XGBModel.get_num_boosting_roundsxs'+'8'8'@#WdFWFWWr4cHt|ds td|jS)Nr2zT`_estimator_type` undefined. Please use appropriate mixin to define estimator type.)r TypeErrorr2rs r5 _get_typezXGBModel._get_type|s.t./I ###r4fnameci}|j|d<tj|}|jj ||jj ||jj dy)Nr2 scikit_learn)r+jsondumpsrset_attr save_model)rIr,rmeta_strs r5r3zXGBModel.save_modelsr!"&.."2 ::d# ###: %%e, ###6r4c|jstd|ji|_|j j ||j j d}|[tj|}|jdd}|2||jk7rtd|jd||j j|_ |j jdtj|j j}|j|y)Nrgr/r2z5Loading an estimator with different type. Expecting: z, got: r.)rrrgrr load_modelattrr0loadsrr+r*rr2 save_config_load_model_attributes)rIr,r4rtconfigs r5r6zXGBModel.load_models))+#Xt{{$;:  ::h'D*D1A}dnn&6!6K~~'(s4 "--/== ###6D,,.::<= ##F+r4r<c:ddlm}|j}|ddd|_|ddd|_|ddd|_|j |_||rrrrrrry n_classes_)rIr<r>rs r5r:zXGBModel._load_model_attributess.""$ *;7?i();._deprecateds. MMI;++4+6  r4c td|d)Nz 2 different `zD` are provided. Use the one in constructor or `set_params` instead.)rTrLs r5 _duplicatedz,XGBModel._configure_fit.._duplicateds  {+++ r4r-rr<exactzYExperimental support for categorical data is not implemented for current tree method yet.)rrrrdrr!r+rrgrfrrrrT)rIrrrrMrOmetricr<s r5_configure_fitzXGBModel._configure_fits gx (3:3F3F3HEE 3 4  3 4 $(    '(()>>#x/1$2B2BDKKPF.t/?/?@F }d.>.>?@jj5  " "{g'=, ff$$r4rc t|jr5|jdk7r& tdi|||j|j dStdi|d|jiS#t $rY$wxYw)Ngblinear)rnthreadrrUr3)rAr<rr"rgrr*r)rIrrs r5_create_dmatrixzXGBModel._create_dmatrixsv (( )dllj.H &"%t{{DLL 5555  s$A$$ A0/A0 evals_resultcv|r7tttttttff||_yyrH)rrrdrre evals_result_rIrWs r5_set_evaluation_resultzXGBModel._set_evaluation_results1 !%d3S$u+5E0F+F&G!VD  r4rFrrverbose xgb_modelrrrrrrFrrr]r^rrrcvt|j5i} tdid|jd|d|ddddd|d |d | d |d |d | ddddd|jd|j d|j \} } |j}t|jrt|j}d|d<nd}|j||\}}}t|| |j| |j| |||||j |_|j#| |cdddS#1swYyxYw)a Fit gradient boosting model. Note that calling ``fit()`` multiple times will cause the model object to be re-fit from scratch. To resume training from a previous checkpoint, explicitly pass ``xgb_model`` argument. Parameters ---------- X : Feature matrix. See :ref:`py-data` for a list of supported types. When the ``tree_method`` is set to ``hist``, internally, the :py:class:`QuantileDMatrix` will be used instead of the :py:class:`DMatrix` for conserving memory. However, this has performance implications when the device of input data is not matched with algorithm. For instance, if the input is a numpy array on CPU but ``cuda`` is used for training, then the data is first processed on CPU then transferred to GPU. y : Labels sample_weight : instance weights base_margin : Global bias for each instance. See :doc:`/tutorials/intercept` for details. eval_set : A list of (X, y) tuple pairs to use as validation sets, for which metrics will be computed. Validation metrics will help us track the performance of the model. verbose : If `verbose` is True and an evaluation set is used, the evaluation metric measured on the validation set is printed to stdout at each boosting stage. If `verbose` is an integer, the evaluation metric is printed at each `verbose` boosting stage. The last boosting stage / the boosting stage found by using `early_stopping_rounds` is also printed. xgb_model : file name of stored XGBoost model or 'Booster' instance XGBoost model to be loaded before training (allows training continuation). sample_weight_eval_set : A list of the form [L_1, L_2, ..., L_n], where each L_i is an array like object storing instance weights for the i-th validation set. base_margin_eval_set : A list of the form [M_1, M_2, ..., M_n], where each M_i is an array like object storing base margin for the i-th validation set. feature_weights : Weight for each feature, defines the probability of each feature being selected when colsample is being used. All values must be greater than 0, otherwise a `ValueError` is thrown. rrrrrNrrFrrrrrrrrrrreg:squarederrorrrr6rWobj custom_metric verbose_evalr^r7r3)rrrrrVrrrr!rr^rRr*r(r6r7rr[)rIrrrFrrr]r^rrrrWrrrrcrrQs r5fitz XGBModel.fitsBdnn 568L#<$ $$$ $  $ , $($!0$"$(>$&:$ $$ $33$$(#:#:$ #00!$ M5$((*F'+?+O&8{#$($7$7 6$J !E66!,,.&*&@&@)$$.. DM  ' ' 5Y6 5 5s DD//D8c$|jdk7ryy)NrTTFrrs r5_can_use_inplace_predictz!XGBModel._can_use_inplace_predictes <<: %r4iteration_rangec||ddk(r d|jdzf}|jdk(rd}|S#t$rd}Y wxYw)Nrr)rrrT)best_iterationAttributeErrorr)rIrjs r5_get_iteration_rangezXGBModel._get_iteration_rangejs_  "oa&8A&= )#$d&9&9A&="> <<: %$O " )"( )s 0 >> output_marginvalidate_featuresc 0t|j5|j|}|jr_ |j j |||rdnd|j ||}t|rddl}|j|}|cdddSt|||j |j|j|j}|j j||||cdddS#t$rYpwxYw#1swYyxYw) aPredict with `X`. If the model is trained with early stopping, then :py:attr:`best_iteration` is used automatically. The estimator uses `inplace_predict` by default and falls back to using :py:class:`DMatrix` if devices between the data and the estimator don't match. .. note:: This function is only thread safe for `gbtree` and `dart`. Parameters ---------- X : Data to predict with. output_margin : Whether to output the raw untransformed margin value. validate_features : When this is True, validate that the Booster's and data's feature_names are identical. Otherwise, it is assumed that the feature_names are the same. base_margin : Global bias for each instance. See :doc:`/tutorials/intercept` for details. iteration_range : Specifies which layer of trees are used in prediction. For example, if a random forest is trained with 100 rounds. Specifying ``iteration_range=(10, 20)``, then only the forests built during [10, 20) (half open set) rounds are used in this prediction. .. versionadded:: 1.4.0 Returns ------- prediction r`marginr )rrj predict_typerrrprN)rrrUrr)rrjrorp)rrrnririnplace_predictrr(cupyasnumpyr*rrgrrpredict) rIrrorprrjpredtsrutests r5rwzXGBModel.predictwsNdnn 5"77HO,,.!--/??(71>XG $ $/*; @F&f-#!%f!5!!6 5*'  "00#'#:#: D##%-- /+"3 .;6 5"!#6 5s0"D AC=AD = D D D  D  Dct|j5|j|}t||j|j |j }|jj|d|cdddS#1swYyxYw)aReturn the predicted leaf every tree for each sample. If the model is trained with early stopping, then :py:attr:`best_iteration` is used automatically. Parameters ---------- X : array_like, shape=[n_samples, n_features] Input features matrix. iteration_range : See :py:meth:`predict`. Returns ------- X_leaves : array_like, shape=[n_samples, n_trees] For each datapoint x in X and for each tree, return the index of the leaf x ends up in. Leaves are numbered within ``[0; 2**(self.max_depth+1))``, possibly with gaps in the numbering. r`)rrrUT) pred_leafrjN) rrrnrrrrgrrw)rIrrj test_dmatrixs r5applyzXGBModel.applysu0dnn 5"77HO" "00 L ##%--o.6 5 5s A BB cNt|dd|j}|Std)aReturn the evaluation results. If **eval_set** is passed to the :py:meth:`fit` function, you can call ``evals_result()`` to get evaluation results for all passed **eval_sets**. When **eval_metric** is also passed to the :py:meth:`fit` function, the **evals_result** will contain the **eval_metrics** passed to the :py:meth:`fit` function. The returned evaluation result is a dictionary: .. code-block:: python {'validation_0': {'logloss': ['0.604835', '0.531479']}, 'validation_1': {'logloss': ['0.41965', '0.17686']}} Returns ------- evals_result rYNz=No evaluation result, `eval_set` is not used during training.)getattrrYr#rZs r5rWzXGBModel.evals_results8* 4$ / ;--L  O r4cB|j}|jS)z.Number of features seen during :py:meth:`fit`.)r num_features)rIrs r5n_features_in_zXGBModel.n_features_in_s!""$##%%r4cz|jj}| tdtj|S)z~Names of features seen during :py:meth:`fit`. Defined only when `X` has feature names that are all strings. zT`feature_names_in_` is defined only when `X` has feature names that are all strings.)r feature_namesrmr\array)rIrs r5feature_names_in_zXGBModel.feature_names_in_ sA ((*88   # xx &&r4c6|jjS)z*The best score obtained by early stopping.)r best_scorers r5rzXGBModel.best_scores!,,,r4c6|jjS)zThe best iteration obtained by early stopping. This attribute is 0-based, for instance if the best iteration is the first round, then best_iteration is 0. )rrlrs r5rlzXGBModel.best_iterations !000r4cj}dtffd }|jjr jn|}|j&t j Dcgc]}d| }}n |j}|Dcgc]}|j|d}}tj|tj}|j} | dk(r|S|| z Scc}wcc}w)auFeature importances property, return depends on `importance_type` parameter. When model trained with multi-class/multi-label/multi-target dataset, the feature importance is "averaged" over all targets. The "average" is defined based on the importance type. For instance, if the importance type is "total_gain", then the score is sum of loss change for each split from all trees. Returns ------- feature_importances_ : array of shape ``[n_features]`` except for multi-class linear model, which returns an array with shape `(n_features, n_classes)` r8c*jdk(rdSdS)NrTrbgainrhrsr5dftz*XGBModel.feature_importances_..dft6s#||z98 Ev Er4)rrgdtyper) rrd get_scorerrr|rrr\rfloat32sum) rIbrscorerlrr all_featuresall_features_arrtotals ` r5feature_importances_zXGBModel.feature_importances_%s%%' FS F 484H4HD00ce  ?? ""*%Xhsm5L0L*M!N?Z!@"#AZ!B CZ!D&d^EZ!F!GZ!H -IZ!J$C=KZ!L$C=MZ!N! OZ!PeCcH$<=>QZ!R (}SZ!TD!123UZ!VWZ!X YZ!xDdO)t) W 3:>ttCH~6S#X,XX$3$7c2;;&6 77D7&00889J,,D,*&00889JLTLdL".%%S 89:.%S#X.% wZ/01 S#X   .%` 68G#4 6 6 6W3C3L3LWQUW .2+/DH.2?C@D>B/3l l l  * l i( l8E)Y*>$?@Al%c *+lE'3 ":;<l!)))< =l'x ':;l"),l l l\$ '7    $"&+/48 I II I i( I ".1 I I\59" "".1"  "Hd3S$u+-=(>#>?<&&&  '2:: ' '-E--111 (bjj ( (Drzz: BJJ  r4rPredtT)boundr predictionvstackc.t|jdksJt|jdk(r|jd|k(r|St|jdk(r|dk(r|jd|k\r|S|}d|z }|||fjS)Nrkrrn)rr transpose)rrrclassone_probsclasszero_probss r5_cls_predict_probars z A %% % :  !j&6&6q&9Y&F J  " N   Q 9 ,NN*O ?N3 4 > > @@r4zBImplementation of the scikit-learn API for XGBoost classification.rrzE n_estimators : Optional[int] Number of boosting rounds. )rc>eZdZedddededdffdZdeee fffd Z edddd ddddd d e d e d e e de e de e ee e fde ee efde eeeefde e e de e e de e ddfdZej&j(Jej&j(j+ddde_ dd e de de de e de ede f fd Z d d e de de e de edej2f fd Zedej2fdZxZS)! XGBClassifierbinary:logisticrrrr8Nc (t|dd|i|yNrr3rrrIrrrs r5rzXGBClassifier.__init__s 7977r4c.t|}d|d<|S)NT multilabelrrrIrrs r5rzXGBClassifier._more_tagssw!#!\ r4Tr\rrrFrrr]r^rrrct|j5i} t|s t|rKddl} | j |j } t| |_| j|j}nt|rAddl} | j |} t| |_| j|j}nDtj tj|} t| |_|j}| j|jk7s| |k(jst!d|d| |j#}t%|j&rt)|j&}d|d<nd}|jdkDr)|j+dddk7rd |d<|j|d <|j-||\}}}t/did |j0d |d |ddddd|d|d| d|d|d| ddddd|j2d|j4d|j6\}}t9|||j;||j<| |||||j> |_ t%|j&s |d|_|jC| |cdddS#1swYyxYw)Nr`rz?Invalid classes inferred from unique values of `y`. Expected: rrrrk multi:softmaxzmulti:softprobrBrrrrrrFrrrrrrrrrrrbr3)"rrr&r'ruuniquevaluesrrDrclasses_r(r\asarrayrrrTrr!rr^rrRrrrVrrr*r(r6r7rr[)rIrrrFrrr]r^rrrrWrclassesexpected_classesrrcrrQrrs r5rfzXGBClassifier.fits dnn 568L1~a!))AHH-"%g,#%88DMM#: "!))A,"%g,#%88DMM#: ))BJJqM2"%g,#'==  !1!7!77#3388: !!1 2& C ((*F'+?+O&7{#"::k40OC*:F;'&*oo{#$($7$7 6$J !E66#<$ $$$ $  $ , $($!0$"$(>$&:$ $$ $33$$(#:#:$ #00!$ M5&",,.&*&@&@)$$.. DMDNN+!' !4  ' ' 5g6 5 5s JJ44J=zFit gradient boosting modelz Fit gradient boosting classifierrrorprjct|j5t| |||||}|r |cdddSt |j dkDr'|j dk7rtj|d}nt |j dkDr:|j ddk7r(tj|j }d||dkD<nb|jdk(r(|jtjcdddStjd|j d}d||dkD<|cdddS#1swYyxYw) Nr`)rrorprrjrrkaxisg?rr)rrrrwrrrDr\argmaxzerosrastyperrepeat) rIrrorprrj class_probscolumn_indexesrs r5rwzXGBClassifier.predictsdnn 5'/+"3' / *K"6 5;$$%)doo.B-/YY{-K[&&'!+ 0A0A!0D0I!#+*;*;!<45{S01?2"))"((3)6 5."$1k.?.?.B!C45{S01!56 5 5sD8B>D8,D88Ec|jdk(r#t| ||||d}t|d}|St| ||||}t |j |t jS)aPredict the probability of each `X` example being of a given class. If the model is trained with early stopping, then :py:attr:`best_iteration` is used automatically. The estimator uses `inplace_predict` by default and falls back to using :py:class:`DMatrix` if devices between the data and the estimator don't match. .. note:: This function is only thread safe for `gbtree` and `dart`. Parameters ---------- X : Feature matrix. See :ref:`py-data` for a list of supported types. validate_features : When this is True, validate that the Booster's and data's feature_names are identical. Otherwise, it is assumed that the feature_names are the same. base_margin : Global bias for each instance. See :doc:`/tutorials/intercept` for details. iteration_range : Specifies which layer of trees are used in prediction. For example, if a random forest is trained with 100 rounds. Specifying `iteration_range=(10, 20)`, then only the forests built during [10, 20) (half open set) rounds are used in this prediction. Returns ------- prediction : a numpy array of shape array-like of shape (n_samples, n_classes) with the probability of each data example being of a given class. rT)rrprrjrorr)rrprrj)rrrwrrrDr\r) rIrrprrj raw_predt class_probrrs r5 predict_probazXGBClassifier.predict_proba8sT >>_ ,"3' /" (I!3J go/#+ & "$//; JJr4c@tj|jSrH)r\arangerDrs r5rzXGBClassifier.classes_tsyy))r4r)TNN)r.r/r0r$rrrrrdrrrr r r rryrrrfr1replacerrwr\r]rrrrrs@r5rrsT #48 8 8  8 8DdO  .2+/DH.2=A@D>B/3b b b  * b i( b8E)Y*>$?@Ab%c *+bE'3"89:b!)))< =b'x ':;b"),b b bH <<   ++ +,,&&..%'I1CK$"&+/48 "" """" "" i( "" ".1 "" ""N#'+/48 :K :K :Ki( :K ".1 :K  :Kx*"****r4rz:scikit-learn API for XGBoost random forest classification.zS n_estimators : Optional[int] Number of trees in random forest to fit. c(eZdZeddddddedededed ef fd Zd eeefffd Z d e fd Z edddddddddde de de e de e de eee e fde eee fde eeeefde ee de ee de e d dffdZxZS)XGBRFClassifierrn皙?h㈵>rrrrrrrrrc nt|d||||d|t|j|jyNrr3rrr;r6r7rIrrrrrrs r5rzXGBRFClassifier.__init__C  '-!     455t~~Fr4r8cFt|}t| |d<|SNrrrr(rIrrs r5rzXGBRFClassifier.get_xgb_params(')&+g&E&G"# r4cyNrr3rs r5r(z'XGBRFClassifier.get_num_boosting_roundsr4NTr\rrrFrrr]r^rrrctjD cic] \} } | dvs | | } } } t|j|jt |di| |Scc} } wN)rIrr3localsrr;r6r7rrfrIrrrFrrr]r^rrrr$r%argsrs r5rfzXGBRFClassifier.fitb"(!1T!1AQ>S5S1!1T455t~~F  d U A#A#r.r/r0r$rerrrrdrryr(rr r r rrrrrfrrs@r5rrysn  #"% GG G  G  GG G$S#X  .2+/DH.2=A@D>B/3    *  i( 8E)Y*>$?@A%c *+E'3"89:!)))< ='x ':;"),  r4rz>Implementation of the scikit-learn API for XGBoost regression.cVeZdZedddededdffdZdeee fffd Z xZ S) XGBRegressorrarrrr8Nc (t|dd|i|yrrrs r5rzXGBRegressor.__init__s 7977r4c8t|}d|d<d|d<|S)NT multioutputFmultioutput_onlyrrs r5rzXGBRegressor._more_tagss*w!#"]#(   r4) r.r/r0r$rrrrrdrrrrs@r5rrsK  +=8(8IL8 8 8 DdOr4rz6scikit-learn API for XGBoost random forest regression.c,eZdZeddddddedededed ed d f fd Zd eeefffd Z d e fdZ ed d d dd d d d dde de de e de e de eee e fde eee fde eeeefde ee de ee de e d dffdZxZS)XGBRFRegressorrnrrrrrrrrr8Nc nt|d||||d|t|j|jyrrrs r5rzXGBRFRegressor.__init__rr4cFt|}t| |d<|Srrrs r5rzXGBRFRegressor.get_xgb_paramsrr4cyrr3rs r5r(z&XGBRFRegressor.get_num_boosting_roundsrr4Tr\rrrFrrr]r^rrrctjD cic] \} } | dvs | | } } } t|j|jt |di| |Scc} } wrrrs r5rfzXGBRFRegressor.fitrrrrs@r5rrsy  #"% GG G  G  GG G G$S#X  .2+/DH.2=A@D>B/3    *  i( 8E)Y*>$?@A%c *+E'3"89:!)))< ='x ':;"),  r4rct|s t|r` for an introducion. raE .. note:: A custom objective function is currently not supported by XGBRanker. .. note:: Query group information is only required for ranking training but not prediction. Multiple groups can be predicted on a single call to :py:meth:`predict`. When fitting the model with the `group` parameter, your data need to be sorted by the query group first. `group` is an array that contains the size of each query group. Similarly, when fitting the model with the `qid` parameter, the data should be sorted according to query index and `qid` is an array that contains the query index for each training sample. For example, if your original data look like: +-------+-----------+---------------+ | qid | label | features | +-------+-----------+---------------+ | 1 | 0 | x_1 | +-------+-----------+---------------+ | 1 | 1 | x_2 | +-------+-----------+---------------+ | 1 | 0 | x_3 | +-------+-----------+---------------+ | 2 | 0 | x_4 | +-------+-----------+---------------+ | 2 | 1 | x_5 | +-------+-----------+---------------+ | 2 | 1 | x_6 | +-------+-----------+---------------+ | 2 | 1 | x_7 | +-------+-----------+---------------+ then :py:meth:`fit` method can be called with either `group` array as ``[3, 4]`` or with `qid` as ``[1, 1, 1, 2, 2, 2, 2]``, that is the qid column. Also, the `qid` can be a special column of input `X` instead of a separated parameter, see :py:meth:`fit` for more info.)rc eZdZedddedeffdZdeede de ded ef fd Z ed d d d d d d d d d d d d de de dee dee dee dee dee e e e fdee e dee e dee eefdee eeefdee e dee e dee d dfdZ d"de dededee deed e f fd Z d#de deed e ffd Zde de d efd!ZxZS)$ XGBRankerz rank:ndcgrrrc t|dd|i|t|jr t dd|vr t dy)Nrz4custom objective function not supported by XGBRankerzrank:z%please use XGBRanker for ranking taskr3)rrr!rrTrs r5rzXGBRanker.__init__GsI 7977 DNN #ST T ) #DE E $r4rrrr8c t||\}}|jdd | tdt|d|||d|S)Nrz4Either `group` or `qid` is required for ranking task)rrrr3)rrrTrrV)rIrrrrrs r5_create_ltr_dmatrixzXGBRanker._create_ltr_dmatrixOsRT3' c ::gt $ ,ST Tw&M3TsMfMMr4NF) rrrFrrrrr]r^rrrrrrrFrrrrr]r^rrrc $t|j5tdid|jd|d|d|d|d|d|d |d |d | d | d |d| d|jd|j d|j \}}i}|j}|j| |\}}}t|||j|j|||| ||j |_ |d|_|j||cdddS#1swYyxYw)aFit gradient boosting ranker Note that calling ``fit()`` multiple times will cause the model object to be re-fit from scratch. To resume training from a previous checkpoint, explicitly pass ``xgb_model`` argument. Parameters ---------- X : Feature matrix. See :ref:`py-data` for a list of supported types. When this is a :py:class:`pandas.DataFrame` or a :py:class:`cudf.DataFrame`, it may contain a special column called ``qid`` for specifying the query index. Using a special column is the same as using the `qid` parameter, except for being compatible with sklearn utility functions like :py:func:`sklearn.model_selection.cross_validation`. The same convention applies to the :py:meth:`XGBRanker.score` and :py:meth:`XGBRanker.predict`. +-----+----------------+----------------+ | qid | feat_0 | feat_1 | +-----+----------------+----------------+ | 0 | :math:`x_{00}` | :math:`x_{01}` | +-----+----------------+----------------+ | 1 | :math:`x_{10}` | :math:`x_{11}` | +-----+----------------+----------------+ | 1 | :math:`x_{20}` | :math:`x_{21}` | +-----+----------------+----------------+ When the ``tree_method`` is set to ``hist``, internally, the :py:class:`QuantileDMatrix` will be used instead of the :py:class:`DMatrix` for conserving memory. However, this has performance implications when the device of input data is not matched with algorithm. For instance, if the input is a numpy array on CPU but ``cuda`` is used for training, then the data is first processed on CPU then transferred to GPU. y : Labels group : Size of each query group of training data. Should have as many elements as the query groups in the training data. If this is set to None, then user must provide qid. qid : Query ID for each training sample. Should have the size of n_samples. If this is set to None, then user must provide group or a special column in X. sample_weight : Query group weights .. note:: Weights are per-group for ranking tasks In ranking task, one weight is assigned to each query group/id (not each data point). This is because we only care about the relative ordering of data points within each group, so it doesn't make sense to assign weights to individual data points. base_margin : Global bias for each instance. See :doc:`/tutorials/intercept` for details. eval_set : A list of (X, y) tuple pairs to use as validation sets, for which metrics will be computed. Validation metrics will help us track the performance of the model. eval_group : A list in which ``eval_group[i]`` is the list containing the sizes of all query groups in the ``i``-th pair in **eval_set**. eval_qid : A list in which ``eval_qid[i]`` is the array containing query ID of ``i``-th pair in **eval_set**. The special column convention in `X` applies to validation datasets as well. verbose : If `verbose` is True and an evaluation set is used, the evaluation metric measured on the validation set is printed to stdout at each boosting stage. If `verbose` is an integer, the evaluation metric is printed at each `verbose` boosting stage. The last boosting stage / the boosting stage found by using `early_stopping_rounds` is also printed. xgb_model : file name of stored XGBoost model or 'Booster' instance XGBoost model to be loaded before training (allows training continuation). sample_weight_eval_set : A list of the form [L_1, L_2, ..., L_n], where each L_i is a list of group weights on the i-th validation set. .. note:: Weights are per-group for ranking tasks In ranking task, one weight is assigned to each query group (not each data point). This is because we only care about the relative ordering of data points within each group, so it doesn't make sense to assign weights to individual data points. base_margin_eval_set : A list of the form [M_1, M_2, ..., M_n], where each M_i is an array like object storing base margin for the i-th validation set. feature_weights : Weight for each feature, defines the probability of each feature being selected when colsample is being used. All values must be greater than 0, otherwise a `ValueError` is thrown. r`rrrrrrFrrrrrrrrrr)num_boost_roundr6rrWrdrer^r7rNr3)rrrrr rrrrRr*r(r6r7rrr[)rIrrrrrFrrrrr]r^rrrrrrWrrrQs r5rfz XGBRanker.fitYsifdnn 5#<$ $$$ $  $ , $($!0$"$(>$&:$&$"$ $77$$(#:#:$ #00!$ M5&79L((*F$($7$7 6$J !E66! $ < < >&*&@&@)$$.. DM$K0DN  ' ' 5S6 5 5s C%DDrorprjcJt|d\}}t| |||||S)N)rj)rrrw)rIrrorprrj_rs r5rwzXGBRanker.predicts:4 1w   +   r4cBt|d\}}t| ||SrH)rrr})rIrrjrrs r5r}zXGBRanker.applys& 4 1w}Q00r4c t|d\}}t||||j|j|j|j }t |jrEt|j|j}|jj|dfg|}n|jj|}t|}|ddS)aEvaluate score for data using the last evaluation metric. If the model is trained with early stopping, then :py:attr:`best_iteration` is used automatically. Parameters ---------- X : Union[pd.DataFrame, cudf.DataFrame] Feature matrix. A DataFrame with a special `qid` column. y : Labels Returns ------- score : The result of the first evaluation metric for the ranker. N)rrrrUreval)fevalrr) rrrrrgrr!rrrrrr%)rIrrrXyqrQ result_str metric_scores r5rzXGBRanker.scores&!T"3 LL#66KK,,  D$$ %)$*:*:DKKHF))+44sFm_F4SJ))+005J&z2 B""r4rrH)r.r/r0r$rdrrr rrr r r rrryrrrfrrwr}rerrrs@r5r r sUj +6FSF#F FNG$N,5NB/3![ [ [  " [ i [ *[i([8E)Y*>$?@A[Xi01[8I./[%c *+[E'3"89:[!)))< =['x ':;[ "),![" #[ [@$"&+/48      i(  ".1    (591 1".11  1&#y&#Y&#5&#r4r )NN)Zr1rr0rzrIconcurrent.futuresrinspectrtypingrrrrr r r r r rrrnumpyr\ scipy.specialr_typingrrrrrcallbackrcompatrrrrr<rcorerrr r!r"r#r$r%rr&r'r(r)trainingr*r,ryr;rdrrArCr]rUrr^rfrrrrerrrrrrrr'rrrrrrrrr r3r4r5r#s 1 1    !SS&YX"   KJ #C= !123   =hsm==*H*I.bjj"**6L0MMN c=,>?@ 3u]L%@A3i3lH$(x(#(6(VSM?# eee3-e!e 3- e ! e}oeeC=/e"e(e_%)e(&)e0#1e0 1e89e8 9eJsm_KeJKeRC=/SeR"Se`sm_ae` aene_ oenoex!(yex)ye@uo&Ae@'AeH%!IeH"IeP }oQeP&Qe`!(ae`)aeh"%)ieh*iep!(qep)qex%!yex"ye@5/"Ae@#AeH!(IeH)IeN5/"OeN#OeVU299#8#8")):M:Ms#RSTUWeVVWer! ser'sez%U4S>3+>%?@A{ezB{eD (c4c 3C.C(DEFEeDGEeTsm_UeT %Uefsm_gefger$D>*ser+seBl+,CeB-CeP"#QeP (Qej"#kej (ke@sm_Ae@ %AeZE#tCy(":;<=[eZ!>[e\&c]O]e\,]eJ$/012KeJ3Kersm_ser se N 2'+" 2 2 92sm2sm 2 tfdl 2jg g  g  g  C= g # g C= g #g c]g xc3h01g %Xc]3g #8C=1g #'g x}%g g g L)!g " 3U38_% %&#g T=(I  |I  I  X  ,A#A6A8APVA$H k\*H/\*\*~@ k0m00fD( 8-   < k0\00f   *  9hy) )*   7*)2fq#.q#g2fq#r4