使用向量数据库向量索引memory超限

SleeeepyZhou · 2025 年5 月 22 日 13:43

【使用环境】测试环境
【 OB or 其他组件】
【使用版本】
【问题描述】(pymysql.err.OperationalError) (7603, "Vector index memory usage exceeds user defined limit '614’M.
【复现路径】
我使用了一个类管理

class RAGDatabase:
    def __init__(self, database: str = "search_rag", dim: int = 4096):
        self.database = database
        if not self._database_create():
            raise Exception(f"[ERROR] 数据库 {self.database} 出错")
        self.client = ObVecClient(
            uri=OCEANBASE_CONFIG["host"] + ":" + str(OCEANBASE_CONFIG["port"]),
            user=OCEANBASE_CONFIG["user"],
            password=OCEANBASE_CONFIG["password"] if "password" in OCEANBASE_CONFIG else "",
            db_name=self.database
        )
        self.dim = dim
        self._table_create()
        self.embedder = EmbeddingModel()
    def insert_data(self, content: str, title: str = None, source: Optional[str] = None, description: Optional[str] = None):

在使用这个类进行了大量数据插入后，报错(pymysql.err.OperationalError) (7603, "Vector index memory usage exceeds user defined limit '614’M.
取消数据存储脚本后，重试发现一开始就报同样错误，是否是因为内存未正确释放？

在重启docker后重试，数据插入正常。

淇铭 · 2025 年5 月 22 日 13:57

pymysql 是哪个版本的

SleeeepyZhou · 2025 年5 月 22 日 14:23

Name: PyMySQL
Version: 1.1.1

SleeeepyZhou · 2025 年5 月 22 日 17:11

你好。还在吗。

淇铭 · 2025 年5 月 22 日 17:20

ob是哪个版本具体的报错信息截图看看

SleeeepyZhou · 2025 年5 月 24 日 09:27

Error inserting 'Finding Nemo (2003) - IMDb' from https://www.imdb.com/title/tt0266543/: (pymysql.err.OperationalError) (7603, "Vector index memory usage exceeds user defined limit '614'M.\n[172.17.0.5:2882] [2025-05-21 12:56:31.942829] [YB42AC110005-0006349E0AECC49F-0-0]")
[SQL: INSERT INTO chunks (section_id, chunk_text, embedding) VALUES (%(section_id_m0)s, %(chunk_text_m0)s, %(embedding_m0)s)]
[parameters: {'section_id_m0': '9780fe1a-bac1-4b10-ba5e-47af84a05811', 'chunk_text_m0': "The story is about Nemo (voiced by Alexander Gould), a young clownfish who is fed up with his dad Marlin's (Albert Brooks) excessive paranoia over him. He swims ...", 'embedding_m0': '[-0.022018433,5.6147575e-05,0.011222839,-0.018539429,0.0046043396,-0.018753052,0.0032215118,0.0075263977,-0.006160736,-0.0002169609,-0.010826111,-0.0 ... (52099 characters truncated) ... 981934,0.0015554428,-0.012649536,0.0023822784,0.0035705566,0.005355835,-0.0013208389,-0.0016345978,0.007637024,0.0032367706,-0.016326904,0.013786316]'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Error inserting 'Finding Nemo: Marlin and Dory' from https://mason.gmu.edu/~hsilvers/375/module4-2-2.html: (pymysql.err.OperationalError) (7603, "Vector index memory usage exceeds user defined limit '614'M.\n[172.17.0.5:2882] [2025-05-21 12:56:32.128119] [YB42AC110005-0006349DFFEAD175-0-0]")
[SQL: INSERT INTO chunks (section_id, chunk_text, embedding) VALUES (%(section_id_m0)s, %(chunk_text_m0)s, %(embedding_m0)s), (%(section_id_m1)s, %(chunk_text_m1)s, %(embedding_m1)s), (%(section_id_m2)s, %(chunk_text_m2)s, %(embedding_m2)s), (%(section_id_m3)s, %(chunk_text_m3)s, %(embedding_m3)s), (%(section_id_m4)s, %(chunk_text_m4)s, %(embedding_m4)s)]
[parameters: {'section_id_m0': '396a34a4-6c7e-4f44-b3e7-092e274cc22b', 'chunk_text_m0': '# Finding Nemo: Marlin and Dory## MarlinMarlin is the main protagonist (main character) in the Disney/Pixar 2003 film\nFinding Nemo .', 'embedding_m0': '[-0.026641846,0.0005350113,0.00289917,0.0069847107,0.00028324127,0.00085926056,0.011253357,0.008964539,0.0010280609,-0.0018129349,0.00868988,-0.00924 ... (52124 characters truncated) ... 9526,-0.0003106594,-0.006298065,-0.004875183,0.023651123,0.0037384033,-0.01586914,-0.0022697449,0.028656006,-0.018707275,-0.0024375916,-0.0015354156]', 'section_id_m1': '396a34a4-6c7e-4f44-b3e7-092e274cc22b', 'chunk_text_m1': 'Marlin is a clownfish who makes his home inside a sea anemone in the Great\nBarrier Reef. He once had a wife Coral who died in a barracuda attack, along\nwith almost all of his children. Also as a result of the attack, he has one\nson, Nemo.', 'embedding_m1': '[-0.017700195,0.012336731,-0.0085372925,0.011291504,-0.0027179718,0.0021896362,0.007232666,-0.006916046,0.0028800964,-0.00010418892,-0.00092697144,-0 ... (52332 characters truncated) ... 6189575,-0.004585266,-0.0015926361,0.006515503,0.008132935,0.008125305,-0.014030457,0.00069379807,0.009178162,-0.006095886,-0.0028572083,0.010612488]', 'section_id_m2': '396a34a4-6c7e-4f44-b3e7-092e274cc22b', 'chunk_text_m2': "Marlin can be somewhat overprotective of Nemo. He also tends to be neurotic,\nworrisome and pessimistic believing Nemo can't swim well when he actually can.\nHowever, this does not stop him from going on a journey to find his lost son,\nproving his determination.", 'embedding_m2': '[-0.02166748,-0.001077652,-0.015174866,0.0073051453,0.019302368,-0.014053345,0.0009851456,-0.0047302246,-0.008399963,0.016967773,-0.016998291,0.00994 ... (51928 characters truncated) ... 515991,-0.003944397,-0.0032100677,0.0037765503,0.022338867,0.0362854,-0.0007581711,-0.0017967224,0.0007185936,0.0059127808,-0.0007786751,0.013641357]', 'section_id_m3': '396a34a4-6c7e-4f44-b3e7-092e274cc22b', 'chunk_text_m3': 'Inspiration for Marlin\'s character came from Sebastian ("The Little Mermaid")\nand Geppetto ("Pinocchio")', 'embedding_m3': '[-0.010192871,0.012237549,0.0029716492,0.012275696,0.0038032532,0.011390686,-0.0013046265,-0.0017728806,-0.022857666,0.00856781,0.014602661,0.0025119 ... (52152 characters truncated) ... .0118255615,0.005619049,0.01574707,0.035888672,0.010627747,0.013175964,0.00944519,0.0019741058,-0.0037879944,-0.0046958923,-0.006137848,-0.013023376]', 'section_id_m4': '396a34a4-6c7e-4f44-b3e7-092e274cc22b', 'chunk_text_m4': 'Marlin is the #17 greatest Disney Hero. [Ultimate Disney.com\n](http://www.dvdizzy.com/disney-heroes-heroines-countdown/)  \n  \nSource: [Marlin](http://disney.wikia.com/wiki/Marlin)', 'embedding_m4': '[-0.025878906,0.017333984,-0.005153656,-0.0063476562,0.028823853,-0.00042510033,0.0075302124,-0.00415802,-0.011856079,0.023483276,-0.010894775,0.0077 ... (52123 characters truncated) ... .022872925,0.0072288513,-0.0034484863,0.0009126663,0.011856079,0.011566162,0.009399414,0.008811951,0.0029582977,-0.016738892,0.005832672,0.004245758]'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

OceanBase 版本: 5.7.25-OceanBase_CE-v4.3.5.1

请问可以一次性把需要的信息问完吗。

君野 · 2025 年5 月 26 日 11:21

https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000002015585 调整下ob_vector_memory_limit_percentage，把向量索引能使用的内存调大一些

淇铭 · 2025 年5 月 26 日 11:25

建议你们发帖的时候版本号错误信息都发一下发帖时候上面写的有规范你们也没有按照规范发帖如果按照规范我们就不用交互那么多次了我们也是一步步排查下次我们也注意

SleeeepyZhou · 2025 年5 月 26 日 21:30

这个调整过，调大之后只会使出现这个错误的时间延后一段时间，但是在插入大量向量数据之后还是会出现这个错误。

乐1983 · 2025 年5 月 27 日 05:04

学习一下

君野 · 2025 年5 月 27 日 15:44

那说明内存还是不够呀，hnsw索引都是需要常驻内存的，数据量大内存分配不够的话，是会报错的，可以参考下参数说明里的内存计算表格，估算下对应的数据量大概需要多少内存。