bcglPddlmZddlmZmZmZmZmZmZm Z ddl m Z ddl m Z ddlZddlZddlZej$Zej&Zej(Zej*Zej,Zej.Zej0Zej2Zej4Zej6Zej8Ze dZdZ dd Zdd Z dd Z!dZ"dZ#ddZ$ddZ%ddZ&ddZ'ddZ(ddddd d dd ddddddddddejRejTfdZ+ddZ,d dZ-y)!) _catboost)Pool CatBoostError ARRAY_TYPES PATH_TYPESfspath_update_params_quantize_part_process_synonyms) defaultdict)contextmanagerNc#K ddlm}|y#t$r.}tjdtt |d}~wwxYww)Nr z,To draw plots you should install matplotlib.)matplotlib.pyplotpyplot ImportErrorwarningswarnstr)pltes I/var/www/html/bid-api/venv/lib/python3.12/site-packages/catboost/utils.py_import_matplotlibrsB"' I " DE#a&!!"s$A A  A)AAA cN|jd|j||dd|jd|jd|j d|j |d|j |d|j|d |jy) N))figsizeg?)alphalwr)fontsizeT) figureplotxticksyticksgridxlabelylabeltitleshow)rxyx_labely_labelr)s r_drawr/!sJJwJHHQH#JJJJJJHHTNJJwJ$JJwJ$IIebI!HHJc $dddddddd} td }tjjD]\}}|j d r||d vrt |t r|g}|D]}t |t s#td jt|||vrtd j||dk(rddg||<a|dk(rddg||<n|dk(rddg||<{ddg||<|dvst |t s#td jt|||vrtd j|| |dg||<| D| jD]1\}}||ddvrtdj||||d<3tt| d5}t|jD]*\}\}}|jdj|||, dddy#1swYyxYw)NLabelWeightBaselineDocIdGroupId SubgroupId Timestamp)labelweightbaselinedoc_idgroup_id subgroup_id timestampc ddgS)NNumrCr0rzcreate_cd..Gsubkr0_) cat_features text_featuresembedding_featuresauxiliary_columnsz,Unsupported index type. Expected int, got {}z"The index {} occurs more than oncerFCategrBrGTextrH NumVector Auxiliary) feature_names output_pathr rArJrKrLzJfeature_names contains index {} that does not correspond to feature columnrwz {} {} {} )r localscopyitems startswith isinstanceintrformattypeopenrsortedwrite)r9rFrGrHr:r;r<r=r>r?rIrNrO_from_param_to_cd_column_descriptionkeyvalueindexfeature_column_indexnamefr)s r create_cdre/sT # &&9:hmmo++- Us#u}bbeS)"GE"E%eS1+,Z,a,abfglbm,noo 33+,P,W,WX],^__n,6=r]+E2/6+D+D+F$G E=E4 GGN))%= >%H ( ' 's 5AHHFc$dddd}|U| tdtt|5}t|j ddj d}dddiig}g}g}gg} g} tt|5}t |D]\} } | j} t| d k(r%| j d} t| d vr td t| d }| d }d}t| d k(r| d}| j|||f ddd| jfd}d}| D]\}}}||k(r td||d z|t| |r|j||}j|gj||dvr|t| z }|d|z}|dk(r|j|d|<ni|dk(r|j|t|<nI|dk(r|j|t|<n)tj|<n| j|||}j||}||d z|t| |||| dS#1swY!xYw#1swYixYw)a Reads CatBoost column description file (see https://catboost.ai/docs/concepts/input-data_column-descfile.html#input-data_column-descfile) Parameters ---------- cd_file : str or pathlib.Path path to column description file column_count : integer total number of columns data_file : str or pathlib.Path path to dataset file in CatBoost format specify either column_count directly or data_file to detect it canonize_column_types : bool if set to True types for columns with synonyms are renamed to canonical type. Returns ------- dict with keys: "column_type_to_indices" : dict of column_type -> column_indices list, column_type is 'Label', 'Categ' etc. "column_dtypes" : dict of column_name -> numpy.dtype or 'category' "cat_feature_indices" : list of integers indices of categorical features in array of all features. Note: indices in array of features, not indices in array of all columns! "text_feature_indices" : list of integers indices of text features in array of all features. Note: indices in array of features, not indices in array of all columns! "embedding_feature_indices" : list of integers indices of embedding features in array of all features. Note: indices in array of features, not indices in array of all columns! "column_names" : list of strings "non_feature_column_indices" : list of integers r2SampleIdr6)Targetr5QueryIdNzjCannot obtain column count: either specify column_count parameter or specify data_file parameter to get it r )rz"Wrong number of columns in cd filerrlrct||D]O}d||z z}j|jdgj|tj|<Qy)N feature_%irA)rangeappend setdefaultnpfloat32)start_column_idxend_column_idxnon_feature_column_countmissed_column_idx column_name column_dtypes column_namescolumn_type_to_indicess radd_missed_columnsz#read_cd..add_missed_columnss`!&'7!H &*;>V*VWK    , " - -eR 8 ? ?@Q R)+M+ & "Ir0z#Duplicate column indices in cd filerPrnrJcategoryrKrL)r{rycat_feature_indicestext_feature_indicesembedding_feature_indicesrznon_feature_column_indices) ExceptionrZrlenreadlinesplit enumeratestriprWrpsortgetrqobjectrrrs)cd_file column_count data_filecanonize_column_typescolumn_type_synonyms_maprdr~rrrcolumn_descriptionsline_idxline line_columns column_idx column_typerxr|last_column_idx feature_idxryrzr{s @@@rread_cdrjs\    ( &# $qzz|CR066t<=L% M "L!# fWo !'lNHd::X:YZ$:' 3!5&?%'A i% $  s,I8BJ8JJrjc t|dkDr+t|dtrtj|n|g}t|dk(rgg}t|dtrtj|n|g}t ||||||||| S)a Evaluate metrics with raw approxes and labels. Parameters ---------- label : list or numpy.ndarrays or pandas.DataFrame or pandas.Series Object labels with shape (n_objects,) or (n_object, n_target_dimension) approx : list or numpy.ndarrays or pandas.DataFrame or pandas.Series Object approxes with shape (n_objects,) or (n_object, n_approx_dimension). metric : string Metric name. weight : list or numpy.ndarray or pandas.DataFrame or pandas.Series, optional (default=None) Object weights. group_id : list or numpy.ndarray or pandas.DataFrame or pandas.Series, optional (default=None) Object group ids. group_weight : list or numpy.ndarray or pandas.DataFrame or pandas.Series, optional (default=None) Group weights. subgroup_id : list or numpy.ndarray, optional (default=None) subgroup id for each instance. If not None, giving 1 dimensional array like data. pairs : list or numpy.ndarray or pandas.DataFrame or string or pathlib.Path The pairs description. If list or numpy.ndarrays or pandas.DataFrame, giving 2 dimensional. The shape should be Nx2, where N is the pairs' count. The first element of the pair is the index of winner object in the training set. The second element of the pair is the index of loser object in the training set. If string or pathlib.Path, giving the path to the file with pairs description. thread_count : int, optional (default=-1) Number of threads to work with. If -1, then the number of threads is set to the number of CPU cores. Returns ------- metric results : list with metric values. r )rrVrrr transpose_eval_metric_util) r9approxmetricr:r= group_weightr>pairs thread_counts r eval_metricrsX 5zA~'1%(K'H U#ug 6{a%/q ;%GR\\& !fXF UFFFHlT_afht uur0c*tjSN)r_get_gpu_device_countrCr0rget_gpu_device_countr6s  * * ,,r0c.tj|yr)r_reset_trace_backend)filenames rreset_trace_backendr:s ""8,r0cft|ts tdt|j||S)a Build confusion matrix. Parameters ---------- model : catboost.CatBoost The trained model. data : catboost.Pool A set of samples to build confusion matrix with. thread_count : int (default=-1) Number of threads to work with. If -1, then the number of threads is set to the number of CPU cores. Returns ------- confusion matrix : array, shape = [n_classes, n_classes] zdata must be a catboost.Pool)rVrr_get_confusion_matrix_object)modeldatars rget_confusion_matrixr>s-( dD !:;; l CCr0c Ht|tr|g}t|ts td|D]}t|trtdt |j ||}|r+t 5}t||d|ddddddd|S|S#1swY|SxYw) a Build points of ROC curve. Parameters ---------- model : catboost.CatBoost The trained model. data : catboost.Pool or list of catboost.Pool A set of samples to build ROC curve with. thread_count : int (default=-1) Number of threads to work with. If -1, then the number of threads is set to the number of CPU cores. plot : bool, optional (default=False) If True, draw curve. Returns ------- curve points : tuple of three arrays (fpr, tpr, thresholds) .data must be a catboost.Pool or list of pools.&one of data pools is not catboost.Poolr rFalse Positive RatezTrue Positive Ratez ROC CurveN)rVrlistr_get_roc_curverrr/)rrrr#pool roc_curvers r get_roc_curverXs.$v dD !LMM$% HI Iu}}dLAI  !S #y|Yq\3HJ^`k l" 9" s 4BB!c l|W| tdt|tst|trt |dk7r td|ddd|ddd}}n || tdt |||\}}}|r't 5}t|||dd d ddd||fS||fS#1swY||fSxYw) a Build points of FPR curve. Parameters ---------- model : catboost.CatBoost The trained model. data : catboost.Pool or list of catboost.Pool A set of samples to build ROC curve with. curve : tuple of three arrays (fpr, tpr, thresholds) ROC curve points in format of get_roc_curve returned value. If set, data parameter must not be set. thread_count : int (default=-1) Number of threads to work with. If -1, then the number of threads is set to the number of CPU cores. plot : bool, optional (default=False) If True, draw curve. Returns ------- curve points : tuple of two arrays (thresholds, fpr) N8Only one of the parameters data and curve should be set.rlCcurve must be list or tuple of three arrays (fpr, tpr, thresholds).r rEmodel and data parameters should be set when curve parameter is None. Thresholdsrz FPR Curve)rrVrtuplerrrr/) rrcurverr#fpr thresholdsrErs r get_fpr_curvers6    Z[ [5$':eU+CE VW ef f(1+uQx{Z =DL gh h*5$ EQ   !S #z3 6K[ Y" s?:s?" s?s B''B3c |T| tdt|tst|trt |dk7r td|d|ddd}}n || tdt |||\}}}t j|Dcgc]}d|z  c}} |r't5} t| || dd d ddd|| fS|| fScc}w#1swY|| fSxYw) a Build points of FNR curve. Parameters ---------- model : catboost.CatBoost The trained model. data : catboost.Pool or list of catboost.Pool A set of samples to build ROC curve with. curve : tuple of three arrays (fpr, tpr, thresholds) ROC curve points in format of get_roc_curve returned value. If set, data parameter must not be set. thread_count : int (default=-1) Number of threads to work with. If -1, then the number of threads is set to the number of CPU cores. plot : bool, optional (default=False) If True, draw curve. Returns ------- curve points : tuple of two arrays (thresholds, fnr) NrrlrrrrrzFalse Negative Ratez FNR Curve) rrVrrrrrrarrayrr/) rrrrr#tprrrEr+fnrrs r get_fnr_curvers6    Z[ [5$':eU+CE VW ef f(E!HQKZ =DL gh h*5$ E3 ((3'3aAE3' (C  !S #z3 6K[ Y" s?:s? (" s?s C )CCc|| td| tdt|tr|g}t|ts td|D]}t|trtdt |j |d|||S|It|tst|t rt|dk7r tdt dd||||Std) a  Selects a threshold for prediction. Parameters ---------- model : catboost.CatBoost The trained model. data : catboost.Pool or list of catboost.Pool Set of samples to build ROC curve with. If set, curve parameter must not be set. curve : tuple of three arrays (fpr, tpr, thresholds) ROC curve points in format of get_roc_curve returned value. If set, data parameter must not be set. FPR : desired false-positive rate FNR : desired false-negative rate (only one of FPR and FNR should be chosen) thread_count : int (default=-1) Number of threads to work with. If -1, then the number of threads is set to the number of CPU cores. Returns ------- threshold : double Nrrrrrlrz3One of the parameters data and curve should be set.)rrVrr_select_thresholdrrr)rrrFPRFNRrrs rselect_thresholdrs:    Z[ [ = gh h dD !6D$% PQ QDdD)#$LMM!dClSS  5$':eU+CE VW ef f tUClKKQRRr0rkc b|s tdt|ts td|t|ts td|t|ts td|t|ts tdi}t|| | } d|vr|j d|d<|j dd}|r(td j |j t|| | | | dd|||||| t||||||||||| }|j|||||||||||| |S) aL Construct quantized Pool from non-quantized pool stored in file. This method does not load whole non-quantized source dataset into memory so it can be used for huge datasets that fit in memory only after quantization. Parameters ---------- data_path : string or pathlib.Path Path (with optional scheme) to non-quantized dataset. column_description : string, [default=None] ColumnsDescription parameter. There are several columns description types: Label, Categ, Num, Auxiliary, DocId, Weight, Baseline, GroupId, Timestamp. All columns are Num as default, it's not necessary to specify this type of columns. Default Label column index is 0 (zero). If None, Label column is 0 (zero) as default, all data columns are Num as default. If string or pathlib.Path, giving the path to the file with ColumnsDescription in column_description format. pairs : string or pathlib.Path, [default=None] Path to the file with pairs description. graph : string or pathlib.Path, [default=None] Path to the file with graph description. has_header : bool, [default=False] If True, read column names from first line. ignore_csv_quoting : bool optional (default=False) If True ignore quoting '"'. feature_names : string or pathlib.Path, [default=None] Path with scheme for feature names data to load. thread_count : int, [default=-1] Thread count for data processing. If -1, then the number of threads is set to the number of CPU cores. ignored_features : list, [default=None] Indices or names of features that should be excluded when training. per_float_feature_quantization : list of strings, [default=None] List of float binarization descriptions. Format : described in documentation on catboost.ai Example 1: ['0:1024'] means that feature 0 will have 1024 borders. Example 2: ['0:border_count=1024', '1:border_count=1024', ...] means that two first features have 1024 borders. Example 3: ['0:nan_mode=Forbidden,border_count=32,border_type=GreedyLogSum', '1:nan_mode=Forbidden,border_count=32,border_type=GreedyLogSum'] - defines more quantization properties for first two features. border_count : int, [default = 254 for training on CPU or 128 for training on GPU] The number of partitions in numeric features binarization. Used in the preliminary calculation. range: [1,65535] on CPU, [1,255] on GPU max_bin : float, synonym for border_count. feature_border_type : string, [default='GreedyLogSum'] The binarization mode in numeric features binarization. Used in the preliminary calculation. Possible values: - 'Median' - 'Uniform' - 'UniformAndQuantiles' - 'GreedyLogSum' - 'MaxLogSum' - 'MinEntropy' nan_mode : string, [default=None] Way to process missing values for numeric features. Possible values: - 'Forbidden' - raises an exception if there is a missing value for a numeric feature in a dataset. - 'Min' - each missing value will be processed as the minimum numerical value. - 'Max' - each missing value will be processed as the maximum numerical value. If None, then nan_mode=Min. input_borders : string or pathlib.Path, [default=None] input file with borders used in numeric features binarization. task_type : string, [default=None] The calcer type used to train the model. Possible values: - 'CPU' - 'GPU' used_ram_limit=None random_seed : int, [default=None] The random seed used for data sampling. If None, 0 is used. Returns ------- pool : Pool Constructed and quantized pool. zData filename is empty.z4Data filename should be string or pathlib.Path type.NzZpairs should have None or string or pathlib.Path type when the pool is read from the file.zgcolumn_description should have None or string or pathlib.Path type when the pool is read from the file.zbfeature_names should have None or string or pathlib.Path type when the pool is read from the file.dev_block_size%dev_max_subset_size_for_build_bordersz'got an unexpected keyword arguments: {}) column_descriptionrgraphrN delimiter has_headerignore_csv_quotingrlog_coutlog_cerr)rr) rrVrr poprXkeysr r_read) data_pathrrrrrrrNrignored_featuresper_float_feature_quantization border_countmax_binfeature_border_typenan_mode input_borders task_typeused_ram_limit random_seedrrkwargsparamsrrs rquantizer sh 566 i ,RSS E:!>xyy%j9KZ.XFG G M:)NAB B Ff 6!#)::.>#? ,2JJ7^`d,e) ELLV[[][\\ &  - -#-! D JJ    Kr0c  ddl}ddl}|j s td|jD]B\}}|dk(r|dvrtjd%|dk(s+|.tjd Dd }|r|j|t }t|j|} |j| } | S#t$r.}tjdtt |d}~wwxYw) a Convert given CatBoost model to ONNX-ML model. Categorical Features are not supported. Parameters ---------- model : CatBoost trained model export_parameters : dict [default=None] Parameters for ONNX-ML export: * onnx_graph_name : string The name property of onnx Graph * onnx_domain : string The domain component of onnx Model * onnx_model_version : int The model_version component of onnx Model * onnx_doc_string : string The doc_string component of onnx Model Returns ------- onnx_object : ModelProto The model in ONNX format r Nz2To get working onnx model you should install onnx.z^There is no trained model to use save_model(). Use fit() to train model. Then use this method. target_opset)NrzUtarget_opset argument is not supported. Default target_opset is 2 (ai.onnx.ml domain) initial_typesz'initial_types argument is not supportedrB)cls)onnxrrrrjson is_fittedrrTdumps_NumpyAwareEncoder_get_onnx_modelrload_model_from_string) rexport_parametersrrrrrcr` params_string model_str onnx_models rconvert_to_onnx_objectrs."  ??  ln n||~ e > !e9&< MMq r _ $): MMC D & M #4:L M  }=I,,Y7J + " JK#a&!!"sB>> C5)C00C5c4|dkDsJdt|||S)Nr zBorder count should be > 0)_calculate_quantization_grid)valuesr border_types rcalculate_quantization_gridrs% ! 999  ' k JJr0) NNNNNNNNNNNNztrain.cd)NNF)NNNNNrj)rj)rjF)NNNrjFr)Median).rBrcorerrrrrr r collectionsr contextlibr sysnumpyrrrrrrrrrrcompute_wx_test TargetStats DataMetaInfocompute_training_optionsrr/rerrrrrrrrrstdoutstderrrrrrCr0rr swwww#% //))!77//11++(EE++## %% $==      8?vUp1vh--D4%P*Z+\/Sh  #'   ZZ ZZ+zz.bKr0