【 使用环境 】生产环境
【 OB or 其他组件 】
obproxy
【 使用版本 】
集群版本详细版本号 (带小版本):obproxy :4.2.5.1-101000092024120918
CPU架构(x86/arm)x86_64
【问题描述】清晰明确描述问题
在进行obproxy节点替换(3节点obproxy集群-> 增加3个节点->删除原有3个节点)后,obproxy内存使用量持续增长,下图是近3天的内存使用量,扩容后开始异常增长
3月7日晚上变更操作步骤如下:
21:46在当前obproxy集群上新增 10.191.205.230 、10.191.205.231两台节点,任务在21:48完成,21:51增加另一个节点10.191.205.232,任务在21:52完成
22:55删除原有集群的3个节点,任务在22:57完成
从ocp监控图上看,3月7日的23点开始obproxy的内存持续增长,到8日15点手动重启obproxy后下降,扩容了obproxy_memory_limit到6G了,目前看还在增长
obproxy扩容前后的版本一致,操作系统的版本一致,obproxy上的连接数、qps变化不大,期间无业务层的变更
【复现路径】问题出现前后相关操作
【附件及日志】
关键日志: 其中ObBufAllocator模块申请了1,074,263,040 实际使用才:426,159,360 疑似泄露
[2026-03-08 14:25:02.638760] ERROR [PROXY] do_monitor_mem (ob_proxy_main.cpp:1097) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] obproxy’s memory is out of limit, will disable alloc memory from the OS(mem_limited=2147483648, OTHER_MEMORY_SIZE=73400320, is_out_of_mem_limit=true, cur_pos=1)
[2026-03-08 14:25:02.638772] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1101) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] history memory size, history_mem_size[0]=2082471936 BACKTRACE:0x247e13 0x22d6aa 0x22f707 0x22fd1d 0x41c357 0xe97f31 0xe96e28 0x14e4bcf 0x14df9e8 0x64513f 0x7fc14d87a14a 0x7fc14d229f23
[2026-03-08 14:25:02.638806] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1101) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] history memory size, history_mem_size[1]=2082471936 BACKTRACE:0x247e13 0x22d6aa 0x22f707 0x22fd1d 0x41c357 0xe97f31 0xe96e28 0x14e4bcf 0x14df9e8 0x64513f 0x7fc14d87a14a 0x7fc14d229f23
[2026-03-08 14:25:02.638814] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1101) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] history memory size, history_mem_size[2]=2082471936 BACKTRACE:0x247e13 0x22d6aa 0x22f707 0x22fd1d 0x41c357 0xe97f31 0xe96e28 0x14e4bcf 0x14df9e8 0x64513f 0x7fc14d87a14a 0x7fc14d229f23
[2026-03-08 14:25:02.638820] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1101) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] history memory size, history_mem_size[3]=2082471936 BACKTRACE:0x247e13 0x22d6aa 0x22f707 0x22fd1d 0x41c357 0xe97f31 0xe96e28 0x14e4bcf 0x14df9e8 0x64513f 0x7fc14d87a14a 0x7fc14d229f23
[2026-03-08 14:25:02.638825] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1101) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] history memory size, history_mem_size[4]=2082471936 BACKTRACE:0x247e13 0x22d6aa 0x22f707 0x22fd1d 0x41c357 0xe97f31 0xe96e28 0x14e4bcf 0x14df9e8 0x64513f 0x7fc14d87a14a 0x7fc14d229f23
[2026-03-08 14:25:02.638830] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1101) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] history memory size, history_mem_size[5]=2082471936 BACKTRACE:0x247e13 0x22d6aa 0x22f707 0x22fd1d 0x41c357 0xe97f31 0xe96e28 0x14e4bcf 0x14df9e8 0x64513f 0x7fc14d87a14a 0x7fc14d229f23
[2026-03-08 14:25:02.638835] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1101) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] history memory size, history_mem_size[6]=2082471936 BACKTRACE:0x247e13 0x22d6aa 0x22f707 0x22fd1d 0x41c357 0xe97f31 0xe96e28 0x14e4bcf 0x14df9e8 0x64513f 0x7fc14d87a14a 0x7fc14d229f23
[2026-03-08 14:25:02.638841] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1101) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] history memory size, history_mem_size[7]=2082471936 BACKTRACE:0x247e13 0x22d6aa 0x22f707 0x22fd1d 0x41c357 0xe97f31 0xe96e28 0x14e4bcf 0x14df9e8 0x64513f 0x7fc14d87a14a 0x7fc14d229f23
[2026-03-08 14:25:02.638845] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1101) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] history memory size, history_mem_size[8]=2082471936 BACKTRACE:0x247e13 0x22d6aa 0x22f707 0x22fd1d 0x41c357 0xe97f31 0xe96e28 0x14e4bcf 0x14df9e8 0x64513f 0x7fc14d87a14a 0x7fc14d229f23
[2026-03-08 14:25:02.638850] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1101) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] history memory size, history_mem_size[9]=2082471936 BACKTRACE:0x247e13 0x22d6aa 0x22f707 0x22fd1d 0x41c357 0xe97f31 0xe96e28 0x14e4bcf 0x14df9e8 0x64513f 0x7fc14d87a14a 0x7fc14d229f23
[2026-03-08 14:25:02.638951] INFO [PROXY] print_memory_usage (ob_proxy_main.cpp:1029) [2415420][Y0-00007FC145610860] [lt=0] [dc=0]
[MEMORY] hold= 35,098,624 used= 34,668,688 count= 2 avg_used= 17,334,344 mod=GLIBC
[2026-03-08 14:25:02.638955] INFO [PROXY] print_memory_usage (ob_proxy_main.cpp:1029) [2415420][Y0-00007FC145610860] [lt=0] [dc=0]
[MEMORY] hold= 0 used= 0 count= 0 avg_used= 0 mod=OB_SQL_AUDIT
[2026-03-08 14:25:02.638956] INFO [PROXY] print_memory_usage (ob_proxy_main.cpp:1029) [2415420][Y0-00007FC145610860] [lt=0] [dc=0]
[MEMORY] hold= 0 used= 0 count= 0 avg_used= 0 mod=OB_SQL_AUDIT_LAST
[2026-03-08 14:25:02.639754] INFO dump (ob_concurrency_objpool.cpp:809) [2415420][Y0-00007FC145610860] [lt=0] [dc=0] dump object freelist statistic:
allocated | in-use | type size | cache type | free list name
--------------------|--------------------|------------|------------|----------------------------------
121,136 | 0 | 1,808 | global | N9oceanbase7obproxy7obutils17ObVCDiagnosisInfoE
594,960 | 590,880 | 240 | global | N9oceanbase7obproxy10prometheus14SQLMonitorInfoE
1,548,288 | 0 | 192 | reclaim | N9oceanbase7obproxy5proxy37ObMysqlRequestCompressTransformPluginE
72,896 | 0 | 1,088 | global | N9oceanbase7obproxy7obutils26ObDetectOneServerStateContE
16,256 | 0 | 128 | global | N9oceanbase7obproxy7obutils28ObServiceNameRoleRefreshContE
8,096 | 3,552 | 32 | global | N9oceanbase7obproxy5proxy24ObProxyPartitionLocationE
24,336 | 22,128 | 48 | global | N9oceanbase7obproxy7obutils14ObSingleLeaderE
40,848 | 4,144 | 592 | global | N9oceanbase7obproxy7obutils24ObTenantSingleLeaderInfoE
16,128 | 12,880 | 112 | global | N9oceanbase6common12LinkHashNodeINS_7obproxy7obutils15ObTenantStatKeyEEE
6,521,856 | 256,304 | 3,088 | reclaim | N9oceanbase7obproxy7obutils16ObTenantStatItemE
619,520 | 154,880 | 9,680 | global | N9oceanbase7obproxy7obutils20ObLocationTenantInfoE
16,160 | 480 | 160 | reclaim | N9oceanbase7obproxy7obutils16ObHashTableEntryINS0_3net12ObIpEndpointEPNS1_17ObCongestionEntryEEE
16,192 | 0 | 176 | global | N9oceanbase7obproxy7obutils21ObCongestRequestParamE
40,656 | 1,584 | 528 | global | N9oceanbase7obproxy7obutils17ObCongestionEntryE
24,320 | 960 | 320 | global | N9oceanbase7obproxy7obutils21ObCongestionZoneStateE
81,120 | 0 | 1,248 | global | N9oceanbase7obproxy5proxy16ObTableEntryContE
8,064 | 96 | 96 | global | N9oceanbase7obproxy7obutils23ObDetectServerStateContE
732,160 | 11,440 | 11,440 | global | N9oceanbase7obproxy7obutils24ObServerStateRefreshContE
3,112,960 | 0 | 304 | reclaim | N9oceanbase7obproxy5proxy22ObTransformVConnectionE
2,058,240 | 0 | 240 | reclaim | N9oceanbase7obproxy5proxy15ObVConnInternalE
6,213,632 | 0 | 656 | reclaim | N9oceanbase7obproxy5proxy38ObMysqlResponseCompressTransformPluginE
8,080 | 0 | 80 | global | N9oceanbase7obproxy5proxy23ObTableCacheHandlerContE
16,128 | 0 | 192 | global | N9oceanbase7obproxy18ObResultSetFetcherE
1,032,192 | 29,824 | 64 | reclaim | N9oceanbase7obproxy5proxy16ObApiTransactionE
18,743,296 | 1,208,064 | 2,288 | reclaim | N9oceanbase7obproxy5proxy20ObMysqlServerSessionE
5,191,680 | 0 | 624 | reclaim | N9oceanbase7obproxy5proxy12ObMysqlRouteE
56,832 | 3,072 | 768 | global | N9oceanbase7obproxy5proxy10ObClientVCE
145,600 | 0 | 2,240 | global | N9oceanbase7obproxy5proxy17ObClientMysqlRespE
32,256 | 0 | 448 | global | N9oceanbase7obproxy5proxy16ObMysqlProxyContE
298,368 | 7,680 | 64 | reclaim | N9oceanbase7obproxy7obutils16ObHashTableEntryINS0_5proxy15ObTableEntryKeyEPNS3_12ObTableEntryEEE
8,112 | 480 | 48 | global | N9oceanbase7obproxy5proxy14ObTenantServerE
440,960 | 6,784 | 6,784 | global | N9oceanbase7obproxy7obutils21ObProxySubClusterInfoE
8,096 | 0 | 368 | global | N9oceanbase7obproxy7obutils27ObClusterResourceCreateContE
114,240 | 1,680 | 1,680 | global | N9oceanbase7obproxy7obutils17ObProxyObInstanceE
32,640 | 384 | 384 | global | N9oceanbase7obproxy7obutils21ObServiceNameInstanceE
72,864 | 1,056 | 1,056 | global | N9oceanbase7obproxy7obutils17ObServiceNameInfoE
121,136 | 0 | 1,808 | global | N9oceanbase7obproxy7obutils28ObProxyInternalDiagnosisInfoE
112,992 | 0 | 1,712 | global | N9oceanbase7obproxy7obutils17ObProxyInsertStmtE
2,023,424 | 1,825,824 | 3,952 | global | N9oceanbase7obproxy5proxy16ObRouteDiagnosisE
202,752 | 70,656 | 3,072 | global | N9oceanbase7obproxy3omt23ObProxyMultiLevelConfigE
24,336 | 22,368 | 48 | global | N9oceanbase7obproxy7obutils26ObConnectionDiagnosisTraceE
17,910,144 | 1,187,264 | 64 | reclaim | N9oceanbase7obproxy5event14ObIOBufferDataE
22,050,320 | 1,518,720 | 80 | reclaim | N9oceanbase7obproxy5event15ObIOBufferBlockE
2,076,672 | 615,888 | 208 | reclaim | N9oceanbase7obproxy5event11ObMIOBufferE
46,923,776 | 5,338,496 | 11,456 | reclaim | N9oceanbase7obproxy5proxy20ObMysqlClientSessionE
11,163,648 | 1,251,360 | 1,264 | reclaim | N9oceanbase7obproxy3net20ObUnixNetVConnectionE
64,512 | 63,616 | 224 | global | N9oceanbase7obproxy10prometheus17ObPrometheusGaugeE
219,648 | 19,968 | 3,328 | global | N9oceanbase7obproxy10prometheus18ObPrometheusFamilyE
16,160 | 10,240 | 80 | global | N9oceanbase7obproxy10prometheus20ObSQLMonitorInfoContE
1,120,896 | 75,824 | 112 | reclaim | N9oceanbase7obproxy5event7ObEventE
1,070,784 | 54,048 | 48 | reclaim | N9oceanbase7obproxy3net9ObEventIOE
132,841,472 | 16,475,456 | 32,432 | reclaim | N9oceanbase7obproxy5proxy9ObMysqlSME
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy39ObMysqlResponseOb20ProtocolGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy41ObMysqlResponsePrepareExecuteGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy33ObMysqlResponseCursorGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy34ObMysqlResponsePrepareGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy35ObMysqlResponseCompressGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy34ObMysqlRequestCompressGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy33ObMysqlRequestExecuteGlobalPluginE
1,038,336 | 384 | 48 | reclaim | N9oceanbase7obproxy5proxy9ObAPIHookE
1,032,192 | 896 | 112 | reclaim | N9oceanbase7obproxy5proxy14ObContInternalE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy33ObMysqlRequestPrepareGlobalPluginE
0 | 0 | 64 | reclaim | N9oceanbase7obproxy5event14ObIOBufferDataE
0 | 0 | 80 | reclaim | N9oceanbase7obproxy5event15ObIOBufferBlockE
0 | 0 | 208 | reclaim | N9oceanbase7obproxy5event11ObMIOBufferE
8,064 | 128 | 64 | global | N9oceanbase7obproxy7obutils25ObCongestionControlConfigE
8,112 | 6,144 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_3net12ObIpEndpointEPNS1_17ObCongestionEntryEEE
114,400 | 58,080 | 880 | global | N9oceanbase7obproxy5proxy13ObMysqlClientE
8,064 | 192 | 96 | global | N9oceanbase7obproxy5proxy17ObMysqlClientPoolE
2,020,608 | 449,024 | 224,512 | global | N9oceanbase7obproxy7obutils17ObClusterResourceE
580,608 | 0 | 9,072 | global | N9oceanbase6common20ObConfigCapacityItemE
580,608 | 0 | 9,072 | global | N9oceanbase6common16ObConfigTimeItemE
580,608 | 0 | 9,072 | global | N9oceanbase6common16ObConfigBoolItemE
841,728 | 0 | 13,152 | global | N9oceanbase6common18ObConfigStringItemE
580,608 | 0 | 9,072 | global | N9oceanbase6common15ObConfigIntItemE
1,172,480 | 0 | 18,320 | global | N9oceanbase6common19ObConfigStrListItemE
195,840 | 182,592 | 288 | global | N9oceanbase7obproxy3omt17ObProxyConfigItemE
113,632 | 1,696 | 1,696 | global | N9oceanbase7obproxy7obutils18ObProxyClusterInfoE
45,280 | 22,640 | 22,640 | global | N9oceanbase7obproxy7obutils21ObProxyJsonConfigInfoE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_5proxy18ObSqlTableEntryKeyEPNS3_15ObSqlTableEntryEEE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_5proxy15ObTableEntryKeyEPNS3_14ObRoutineEntryEEE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_4obkv20ObTableApiCredentialEPNS0_5proxy11ObRpcReqCtxEEE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_5proxy20ObTableGroupEntryKeyEPNS3_17ObTableGroupEntryEEE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableImPNS0_5proxy22ObTableQueryAsyncEntryEEE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_5proxy15ObIndexEntryKeyEPNS3_12ObIndexEntryEEE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_5proxy19ObPartitionEntryKeyEPNS3_16ObPartitionEntryEEE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_5proxy15ObTableEntryKeyEPNS3_12ObTableEntryEEE
1,177,344 | 122,240 | 64 | reclaim | N9oceanbase7obproxy5event12ObProxyMutexE
1,074,263,040 | 426,159,360 | 8,208 | reclaim | ObBufAllocator
35,013,680 | 6,015,856 | 4,112 | reclaim | ObBufAllocator
18,115,728 | 4,379,808 | 2,064 | reclaim | ObBufAllocator
9,391,200 | 674,960 | 1,040 | reclaim | ObBufAllocator
5,203,968 | 490,512 | 528 | reclaim | ObBufAllocator
3,133,440 | 50,592 | 272 | reclaim | ObBufAllocator
2,278,080 | 433,296 | 144 | reclaim | ObBufAllocator
2,162,048 | 608,640 | 128 | reclaim | ObThreadCache
48,960 | 25,840 | 272 | global | ObjFreeList


