bcgdQ^ddlZddlZddlmZddlmZddlmZddlZejdk\rddl Z dZ nddl m Z dZd Zed ZGd d eZGd deZeZej(Zej*ej,ej.ej0ej2ej4iZej*ej8ej.ej:ej2ej<iZej*ej@ej.ejBej2ejDiZ#ej*ejHej.ejJej2ejLiZ'GddZ(dZ)GddZ*GddZ+GddZ,y)N) iteritems)IntEnum)contextmanager)ctjj||}tjj|}|jj ||SN) importlibutilspec_from_file_locationmodule_from_specloader exec_module)namepathspecmodules M/var/www/html/bid-api/venv/lib/python3.12/site-packages/catboost/hnsw/hnsw.py load_dynamicr sC~~55dDA006 ' )rctjjtjjt|}tjj |rtj |ng}|Dcgc]9}|jdddvstjj||;c}Scc}w)N.)sopyd)osrjoindirname__file__isdirlistdirsplit)dir_namelist_dirso_names r get_so_pathsr%sww||BGGOOH5x@H')ww}}X'>rzz(#BH;C o8w}}UXGYZ\G]anGnBGGLL7 +8 oo os <B="B=cdtjvrtjdStd}|D]%} td|}|tjd<|cSddlm}|S#t$rY;wxYw)N_hnswz./z hnsw._hnsw)r')sysmodulesr%r ImportErrorr')so_pathsso_path loaded_hnswr's rget_hnsw_bin_moduler0st#++{{7##D!H &w8K(3CKK %   L   s A## A/.A/c#Ktjtj dtj y#tj wxYwwr)r' _set_loggerr)stdout _reset_loggerrr log_fixupr6)s8 cjj!  s$AAAAAceZdZdZdZdZdZy) EDistancerr(rN)__name__ __module__ __qualname__ DotProductL1L2SqrPairVectorDistancer5rrr8r82sJ B Err8ceZdZdZdZdZy)EVectorComponentTyperr(r9N)r:r;r<FloatI8I32r5rrrBrB9s E B CrrBcDeZdZdZddZedZedZdZdZ y) Poolz& Pool is a storage of vectors Nc||_||_||_|du|duz sJ|t||||_d|_|t|d|||_||_yy)a Pool is a storage of vectors. You can create it from row-major binary file or binary data of vectors. Parameters ---------- vectors_path : string Path to binary file with vectors. dtype : EVectorComponentType Type of vectors. dimension : int Dimension of vectors. vectors_bin_data : bytes Binary data of vectors. N) vectors_pathdtype dimension_DenseVectorStorage_storage_data)selfrIrJrKvectors_bin_datas r__init__z Pool.__init__asz&) " D(\T-ABBB  #/6|YODMDJ  '/6tYHXYDM)DJ (rct|||dS)a  Create pool from binary file. Parameters ---------- vectors_path : string Path to binary file with vectors. dtype : EVectorComponentType Type of vectors. dimension : int Dimension of vectors. NrG)clsrIrJrKs r from_filezPool.from_files L%D99rctd|||S)a Create pool from binary data. Parameters ---------- vectors_bin_data : bytes Binary data of vectors. dtype : EVectorComponentType Type of vectors. dimension : int Dimension of vectors. NrS)rTrPrJrKs r from_byteszPool.from_bytess D%,<==rc8|jj|Sz Get item from storage by id. Parameters ---------- id : int Index of item in storage. Returns ------- item : numpy.ndarray )rM _get_itemrOids rget_itemz Pool.get_items}}&&r**rc6|jjSzn Get the number of items in storage. Returns ------- num_items : int )rM_get_num_itemsrOs r get_num_itemszPool.get_num_itemss}}++--rr) r:r;r<__doc__rQ classmethodrUrWr]rbr5rrrGrG\s>*<::">>" +.rrGctjtdtj|j }t |j|j|_|S)a  Transform pool for fast dot product search on HNSW graph https://papers.nips.cc/paper/9032-mobius-transformation-for-fast-inner-product-search-on-graph.pdf Parameters ---------- pool : Pool Returns ------- transformed_pool : Pool r) rGrWbytesrBrCrK_transform_mobiusrJrM)pooltransformed_pools rtransform_mobiusrjsGuQx1E1K1KT^^\ 1$** =dmm L rcDeZdZdZdZ d dZdZdZdZdZ d d Z y) Hnswz` Class for building, loading and working with Hierarchical Navigable Small World index. c d|_d|_y)z6 Create object for working with HNSW. N)_indexrNras rrQz Hnsw.__init__s  rNcJi}gd}ttD]\}}||vs ||||<t|j|j||_t 5|j jtj|dddy#1swYyxYw)a Build index with given options. Parameters ---------- pool : Pool Pool of vectors for which index will be built. distance : EDistance Distance that should be used for finding nearest vectors. max_neighbors : int (default=32) Maximum number of neighbors that every item can be connected with. search_neighborhood_size : int (default=300) Search neighborhood size for ANN-search. Higher values improve search quality in expense of building time. num_exact_candidates : int (default=100) Number of nearest vectors to take from batch. Higher values improve search quality in expense of building time. batch_size : int (default=1000) Number of items that added to graph on each step of algorithm. upper_level_batch_size : int (default=40000) Batch size for building upper levels. level_size_decay : int (default=max_neighbors/2) Base of exponent for decaying level sizes. num_threads : int (default=number of CPUs) Number of threads for building index. report_progress : bool (default=True) Print progress of building. verbose : bool (default=False) Print additional information about time of building. snapshot_file : string (default=None) Path for saving snapshots during the index building. snapshot_interval : int (default=600) Interval between saving snapshots (seconds). Snapshot is saved after building each level also. ) not_paramsrOparams __class__rhdistanceN) rlocals_HnswDenseVectorIndexrJrMrnr6_buildjsondumps)rOrhrs max_neighborssearch_neighborhood_sizenum_exact_candidates batch_sizeupper_level_batch_sizelevel_size_decay num_threadsverbosereport_progress snapshot_filesnapshot_intervalrqrpkeyvalues rbuildz Hnsw.buildsdV #FH-JC*$):#s .,DJJ7 xP [ KK  tzz&1 2[[s !/BB"c2|j tdyNz!Index is not built and not loadedrn HnswExceptionras r _check_indexzHnsw._check_index ;;  CD D rcZ|j|jj|y)z Save index to file. Parameters ---------- index_path : string Path to file for saving index. N)rrn_save)rO index_paths rsavez Hnsw.saves"  *%rct|j|j||_|jj |d|_y)aK Load index from file. Parameters ---------- index_path : string Path to file for loading index. pool : Pool Pool of vectors for which index will be loaded. distance : EDistance Distance that should be used for finding nearest vectors. N)rurJrMrn_loadrN)rOrrhrss rloadz Hnsw.load%s8,DJJ7 xP  *% rct|j|j||_|jj |||_y)a> Load index from bytes. Parameters ---------- index_data : bytes Index binary data. pool : Pool Pool of vectors for which index will be loaded. distance : EDistance Distance that should be used for finding nearest vectors. N)rurJrMrn_load_from_bytesrN)rO index_datarhrss rload_from_byteszHnsw.load_from_bytes8s8,DJJ7 xP  $$Z0 rc^|j|jj||||S)a Get approximate nearest neighbors for query from index. Parameters ---------- query : list or numpy.ndarray Vector for which nearest neighbors should be found. top_size : int Required number of neighbors. search_neighborhood_size : int Search neighborhood size for ANN-search. Higher values improve search quality in expense of search time. It should be equal or greater than top_size. distance_calc_limit : int (default=0) Limit of distance calculation. To guarantee satisfactory search time at the expense of quality. 0 is equivalent to no limit. Returns ------- neighbors : list of tuples (id, distance) )rrn _get_nearest)rOquerytop_sizerzdistance_calc_limits r get_nearestzHnsw.get_nearestKs-4 {{''x9QSfggr) NNNNNNNFTNNr) r:r;r<rcrQrrrrrrr5rrrlrls:mqmrJN93vE && &hrrlceZdZdZdej ddddddfd Zd Z dd Zd Z dd Z dZ e dZ e dZ ddZy) HnswEstimatorz Class for building, loading and working with Hierarchical Navigable Small World index with SciKit-Learn Estimator compatible interface. Mostly drop-in replacement for sklearn.neighbors.NearestNeighbors (except for some parameters)  i,dii@Nc ^ttD]\} } | dvs t|| | y)a< Parameters ---------- n_neighbors : int, default=5 Number of neighbors to use by default for kneighbors queries. distance : EDistance Distance that should be used for finding nearest vectors. max_neighbors : int (default=32) Maximum number of neighbors that every item can be connected with. search_neighborhood_size : int (default=300) Search neighborhood size for ANN-search. Higher values improve search quality in expense of building time. num_exact_candidates : int (default=100) Number of nearest vectors to take from batch. Higher values improve search quality in expense of building time. batch_size : int (default=1000) Number of items that added to graph on each step of algorithm. upper_level_batch_size : int (default=40000) Batch size for building upper levels. level_size_decay : int (default=max_neighbors/2) Base of exponent for decaying level sizes. )rOrrN)rrtsetattr) rO n_neighborsrsryrzr{r|r}r~rrs rrQzHnswEstimator.__init__ps/D$FH-JC//c5).rc2|j tdyrrras rrzHnswEstimator._check_indexrrc~tj||j\|_|_|j d}gd} t tD]\} } | | vs | | || <|d=t5|jjtj|ddd|S#1swY|SxYw)a Fit the HNSW model. Parameters ---------- X : array-like of shape (n_samples, n_values) y: None Added to be compatible with Estimator API num_threads : int (default=number of CPUs) Number of threads for building index. report_progress : bool (default=True) Print progress of building. verbose : bool (default=False) Print additional information about time of building. snapshot_file : string (default=None) Path for saving snapshots during the index building. snapshot_interval : int (default=600) Interval between saving snapshots (seconds). Returns ------- model : HnswEstimator F return_none)rprOrqrrXyNrs) r' _init_indexrsrn _index_data _get_paramsrrtr6rvrwrx) rOrrrrrrrrqrprrs rfitzHnswEstimator.fits@).(9(9!T]](K% T%!!e!4L #FH-JC*$):#s . :  [ KK  tzz&1 2  s 9/B22B<cti}|jjD]\}}|ddk7s|s||||<|S)Nr_)__dict__items)rOrrqrrs rrzHnswEstimator._get_paramssD----/JC1v}+%2C#s 0 rc&|jdS)z4 Get parameters for this estimator. Tr)r)rOdeeps r get_paramszHnswEstimator.get_paramssD11rc |s|S|jd}|jD]\}}||vs td|S)z Set the parameters of this estimator. Parameters ---------- **params : dict HnswEstimator parameters. Returns ------- self : HnswEstimator instance TrzjInvalid parameter %s for HnswEstimator. Check the list of available parameters with `get_params().keys()`.)rrrr)rOrq valid_paramsrrs r set_paramszHnswEstimator.set_paramssTK''D'9  ,,.JC,&#2) rc|jS)zk Returns ------- Distance that should be used for finding nearest vectors. )rsras reffective_metric_zHnswEstimator.effective_metric_s}}rcT|j|jjdS)zW Returns ------- Number of samples in the fitted data. r)rrshaperas rn_samples_fit_zHnswEstimator.n_samples_fit_s' %%a((rc|j| |j}| |j}||dz}|jj ||||j ||S)aFinds the approximate K-neighbors of a point. Returns indices of and distances to the neighbors of each point. Parameters ---------- X : array-like, shape (n_queries, n_features) or None The query point or points. If not provided, neighbors of each indexed point are returned. In this case, the query point is not considered its own neighbor. n_neighbors : int, default=None Number of neighbors required for each sample. The default is the value passed to the constructor. return_distance : bool, default=True Whether or not to return the distances. search_neighborhood_size : int, default=None Search neighborhood size for ANN-search. Higher values improve search quality in expense of search time. It should be equal or greater than top_size. If None set to n_neighbors * 2. distance_calc_limit : int (default=0) Limit of distance calculation. To guarantee satisfactory search time at the expense of quality. 0 is equivalent to no limit. Returns ------- neigh_dist :numpy.ndarray of shape (n_queries, n_neighbors) Array representing the lengths to points, only present if return_distance=True neigh_ind : numpy.ndarray of shape (n_queries, n_neighbors) Indices of the nearest points in the population matrix. r9)rrrrn _kneighborsrs)rOrrreturn_distancerzrs r kneighborszHnswEstimator.kneighborsskH  9  A  **K # +'2Q ${{&&q+ Wo':< M J" + 03rr)-r)rsixrenumr contextlibrrw version_infoimportlib.utilr rimpr%r0r6r8rBr'rrC_DenseFloatVectorStoragerD_DenseI8VectorStoragerE_DenseI32VectorStoragerL_HnswDenseFloatVectorIndex_HnswDenseI8VectorIndex_HnswDenseI32VectorIndexru_transform_mobius_float_transform_mobius_i8_transform_mobius_i32rg _OnlineHnswDenseFloatVectorIndex_OnlineHnswDenseI8VectorIndex_OnlineHnswDenseI32VectorIndexrrGrjrlrrr5rrrs % t !p  7 ##  > >U88e:: @ @U::e<< = =U77e99 F FU@@eBB^.^.B$WhWhtH<H