【 使用环境 】生产环境
【 OB or 其他组件 】obproxy
【 使用版本 】4.2.3.0-3
【问题描述】obproxy 内存超限，修改了 proxy_mem_limited= 60g 还是报错
1 个赞
优先使用obdiag 过滤一下近一小时日志是否存在异常
obdiag analyze log --since 1h
#内存问题
obdiag gather scene run --scene=observer.memory
提供一下odp的日志
1 个赞
集群架构提供一下，odp架构提供一下，odp原来的内存设置是多少，obp修改不是全局的，是不是其他节点obp满了 修的只是其中1个
1 个赞
obproxy跟observer 部署在一块，3zone ，每个zone 1台服务器。
1 个赞
我是从ocp 平台修改的，不会出现 只修改一个节点的吧
1 个赞
收集了日志，传不上去了，安全管控了
1 个赞
odp内存默认设置的就是64G么
紧急处理的话就是重启odp
1 个赞
[2024-08-12 16:10:50.577334] ERROR [PROXY] do_monitor_mem (ob_proxy_main.cpp:1038) [179976][Y0-00002B8DAB3D9350] [lt=0] [dc=0] obproxy's memroy is out of limit's 90% !!!(mem_limited=68719476736, OTHER_MEMORY_SIZE=73400320, is_out_of_error_mem_limit=true, cur_pos=3)
[2024-08-12 16:10:50.579352] INFO [PROXY] print_memory_usage (ob_proxy_main.cpp:978) [179976][Y0-00002B8DAB3D9350] [lt=0] [dc=0]
[MEMORY] hold= 88,252,416 used= 87,948,960 count= 134 avg_used= 656,335 mod=GLIBC
[2024-08-12 16:10:50.579355] INFO [PROXY] print_memory_usage (ob_proxy_main.cpp:978) [179976][Y0-00002B8DAB3D9350] [lt=0] [dc=0]
[MEMORY] hold= 0 used= 0 count= 0 avg_used= 0 mod=OB_SQL_AUDIT
[2024-08-12 16:10:50.579357] INFO [PROXY] print_memory_usage (ob_proxy_main.cpp:978) [179976][Y0-00002B8DAB3D9350] [lt=0] [dc=0]
[MEMORY] hold= 0 used= 0 count= 0 avg_used= 0 mod=OB_SQL_AUDIT_LAST
[2024-08-12 16:10:50.580539] INFO dump (ob_concurrency_objpool.cpp:810) [179976][Y0-00002B8DAB3D9350] [lt=0] [dc=0] dump object freelist statistic:
allocated | in-use | type size | cache type | free list name
--------------------|--------------------|------------|------------|----------------------------------
40,400 | 880 | 80 | reclaim | N9oceanbase7obproxy7obutils16ObHashTableEntryINS0_5proxy19ObPartitionEntryKeyEPNS3_16ObPartitionEntryEEE
80,960 | 1,936 | 176 | reclaim | N9oceanbase7obproxy5proxy16ObPartitionEntryE
24,288 | 0 | 352 | global | N9oceanbase7obproxy5proxy20ObPartitionEntryContE
793,520 | 0 | 1,744 | global | N9oceanbase7obproxy7obutils17ObVCDiagnosisInfoE
113,360 | 0 | 1,744 | global | N9oceanbase7obproxy7obutils28ObProxyInternalDiagnosisInfoE
72,960 | 0 | 256 | reclaim | N9oceanbase7obproxy5proxy37ObMysqlResponsePrepareTransformPluginE
64,512 | 0 | 1,008 | global | N9oceanbase7obproxy7obutils26ObDetectOneServerStateContE
40,480 | 35,424 | 32 | global | N9oceanbase7obproxy5proxy24ObProxyPartitionLocationE
113,360 | 0 | 1,744 | global | N9oceanbase7obproxy7obutils20ObLoginDiagnosisInfoE
5,575,680 | 5,324,000 | 9,680 | global | N9oceanbase7obproxy7obutils20ObLocationTenantInfoE
242,400 | 3,520 | 160 | reclaim | N9oceanbase7obproxy7obutils16ObHashTableEntryINS0_3net12ObIpEndpointEPNS1_17ObCongestionEntryEEE
16,192 | 0 | 176 | global | N9oceanbase7obproxy7obutils21ObCongestRequestParamE
40,656 | 11,616 | 528 | global | N9oceanbase7obproxy7obutils17ObCongestionEntryE
24,320 | 3,200 | 320 | global | N9oceanbase7obproxy7obutils21ObCongestionZoneStateE
129,024 | 125,216 | 112 | global | N9oceanbase6common12LinkHashNodeINS_7obproxy7obutils15ObTenantStatKeyEEE
117,089,280 | 19,480,032 | 17,424 | reclaim | N9oceanbase7obproxy7obutils16ObTenantStatItemE
40,320 | 0 | 576 | global | N9oceanbase7obproxy5proxy16ObTableEntryContE
8,064 | 384 | 96 | global | N9oceanbase7obproxy7obutils23ObDetectServerStateContE
616,448 | 38,528 | 9,632 | global | N9oceanbase7obproxy7obutils24ObServerStateRefreshContE
3,088,640 | 0 | 304 | reclaim | N9oceanbase7obproxy5proxy22ObTransformVConnectionE
2,042,160 | 0 | 240 | reclaim | N9oceanbase7obproxy5proxy15ObVConnInternalE
10,346,944 | 0 | 1,216 | reclaim | N9oceanbase7obproxy5proxy38ObMysqlResponseCompressTransformPluginE
8,080 | 0 | 80 | global | N9oceanbase7obproxy5proxy23ObTableCacheHandlerContE
16,128 | 0 | 192 | global | N9oceanbase7obproxy18ObResultSetFetcherE
1,032,192 | 205,824 | 64 | reclaim | N9oceanbase7obproxy5proxy16ObApiTransactionE
25,952,256 | 14,002,560 | 3,168 | reclaim | N9oceanbase7obproxy5proxy20ObMysqlServerSessionE
87,162,880 | 0 | 10,640 | reclaim | N9oceanbase7obproxy5proxy12ObMysqlRouteE
56,832 | 25,344 | 768 | global | N9oceanbase7obproxy5proxy10ObClientVCE
130,944 | 0 | 1,984 | global | N9oceanbase7obproxy5proxy17ObClientMysqlRespE
32,400 | 0 | 432 | global | N9oceanbase7obproxy5proxy16ObMysqlProxyContE
919,296 | 78,976 | 64 | reclaim | N9oceanbase7obproxy7obutils16ObHashTableEntryINS0_5proxy15ObTableEntryKeyEPNS3_12ObTableEntryEEE
8,112 | 5,520 | 48 | global | N9oceanbase7obproxy5proxy14ObTenantServerE
412,672 | 25,792 | 6,448 | global | N9oceanbase7obproxy7obutils21ObProxySubClusterInfoE
8,096 | 0 | 368 | global | N9oceanbase7obproxy7obutils27ObClusterResourceCreateContE
677,376 | 672,000 | 224 | global | N9oceanbase7obproxy10prometheus17ObPrometheusGaugeE
1,130,496 | 88,320 | 17,664 | global | N9oceanbase7obproxy10prometheus18ObPrometheusFamilyE
11,789,440 | 11,102,304 | 3,488 | global | N9oceanbase7obproxy5proxy16ObRouteDiagnosisE
170,352 | 154,368 | 48 | global | N9oceanbase7obproxy7obutils26ObConnectionDiagnosisTraceE
8,975,232 | 1,095,488 | 64 | reclaim | N9oceanbase7obproxy5event14ObIOBufferDataE
11,134,240 | 1,642,480 | 80 | reclaim | N9oceanbase7obproxy5event15ObIOBufferBlockE
4,688,736 | 3,324,880 | 208 | reclaim | N9oceanbase7obproxy5event11ObMIOBufferE
60,555,264 | 47,545,344 | 14,784 | reclaim | N9oceanbase7obproxy5proxy20ObMysqlClientSessionE
14,723,072 | 9,610,192 | 1,264 | reclaim | N9oceanbase7obproxy3net20ObUnixNetVConnectionE
16,160 | 10,240 | 80 | global | N9oceanbase7obproxy10prometheus20ObSQLMonitorInfoContE
1,112,832 | 59,696 | 112 | reclaim | N9oceanbase7obproxy5event7ObEventE
1,208,688 | 371,088 | 48 | reclaim | N9oceanbase7obproxy3net9ObEventIOE
202,437,120 | 167,373,520 | 49,040 | reclaim | N9oceanbase7obproxy5proxy9ObMysqlSME
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy39ObMysqlResponseOb20ProtocolGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy41ObMysqlResponsePrepareExecuteGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy33ObMysqlResponseCursorGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy34ObMysqlResponsePrepareGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy35ObMysqlResponseCompressGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy34ObMysqlRequestCompressGlobalPluginE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy33ObMysqlRequestExecuteGlobalPluginE
1,030,224 | 384 | 48 | reclaim | N9oceanbase7obproxy5proxy9ObAPIHookE
1,024,128 | 896 | 112 | reclaim | N9oceanbase7obproxy5proxy14ObContInternalE
8,096 | 32 | 32 | reclaim | N9oceanbase7obproxy5proxy33ObMysqlRequestPrepareGlobalPluginE
0 | 0 | 64 | reclaim | N9oceanbase7obproxy5event14ObIOBufferDataE
0 | 0 | 80 | reclaim | N9oceanbase7obproxy5event15ObIOBufferBlockE
0 | 0 | 208 | reclaim | N9oceanbase7obproxy5event11ObMIOBufferE
8,064 | 320 | 64 | global | N9oceanbase7obproxy7obutils25ObCongestionControlConfigE
16,224 | 15,360 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_3net12ObIpEndpointEPNS1_17ObCongestionEntryEEE
286,000 | 227,040 | 880 | global | N9oceanbase7obproxy5proxy13ObMysqlClientE
8,064 | 480 | 96 | global | N9oceanbase7obproxy5proxy17ObMysqlClientPoolE
6,039,040 | 6,039,040 | 1,207,808 | global | N9oceanbase7obproxy7obutils17ObClusterResourceE
5,501,952 | 5,330,016 | 9,552 | global | N9oceanbase7obproxy3omt17ObProxyConfigItemE
113,632 | 10,176 | 1,696 | global | N9oceanbase7obproxy7obutils18ObProxyClusterInfoE
44,192 | 22,096 | 22,096 | global | N9oceanbase7obproxy7obutils21ObProxyJsonConfigInfoE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_5proxy18ObSqlTableEntryKeyEPNS3_15ObSqlTableEntryEEE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_5proxy15ObTableEntryKeyEPNS3_14ObRoutineEntryEEE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_5proxy19ObPartitionEntryKeyEPNS3_16ObPartitionEntryEEE
8,112 | 3,072 | 48 | global | N9oceanbase7obproxy7obutils14ObIMTHashTableINS0_5proxy15ObTableEntryKeyEPNS3_12ObTableEntryEEE
1,330,560 | 289,856 | 64 | reclaim | N9oceanbase7obproxy5event12ObProxyMutexE
689,734,656 | 139,864,320 | 8,208 | reclaim | ObBufAllocator
47,575,840 | 31,469,136 | 4,112 | reclaim | ObBufAllocator
40,103,520 | 31,804,176 | 2,064 | reclaim | ObBufAllocator
9,318,400 | 3,594,240 | 1,040 | reclaim | ObBufAllocator
6,017,088 | 3,489,552 | 528 | reclaim | ObBufAllocator
3,133,440 | 325,040 | 272 | reclaim | ObBufAllocator
3,742,560 | 1,950,912 | 144 | reclaim | ObBufAllocator
2,162,048 | 613,760 | 128 | reclaim | ObThreadCache
24,480 | 21,760 | 272 | global | ObjFreeList
1 个赞
不是，之前设置的4G，改了8g 还是不行，慢慢改到64G了 。
然后 obproxy 在不断重启，在obproxy.log 看到
roxy_main.cpp:1038) [179976][Y0-00002B8DAB3D9350] [lt=0] [dc=0] obproxy's memroy is out of limit's 90% !!!(mem_limited=68719476736, OTHER_MEMORY_SIZE=73400320, is_out_of_error_mem_limit=true, cur_pos=3)
1 个赞
*************************** 28. row ***************************
mod_name: OB_LARGE_IO_BUFFER
mod_type: user
hold: 47,733,342,848
used: 42,429,685,328
count: 2,533
avg_used: 16,750,764
hold bytes, alloc times and backtraces(addr2line -Cfe obproxy 0xXXXX):
1342178560 80
0x1c04aa 0x63f7d8 0x220bc0 0x1bc4ef 0xa58f74 0xa7df9f 0xa6a933 0x7af339 0x7b60bd 0x7c0481 0x7bef9d 0x27de77 0x27f22c 0x23c4ae 0x1528496 0x15243b1
1 个赞
目前是重启也无效，重启后odp会立马继续上涨么
对，
看起来像是 这个版本内存方面有点问题，可以将其中一个节点的obproxy 从obproxy集群中删除，然后部署一个新的obproxy 集群，版本 4.2.1 的最新版本。可以立即对比观察判断是不是版本问题。
生产使用obproxy也是这版本占用内存很低1G，主要TPS写入高场景，你们使用是偏AP查询类多吗
/home/admin/obproxy/bin/obproxy --version
obproxy (OceanBase 4.2.3.0 3)
问题找到了，改了一个obproxy 的参数，导致在大并发的场景下 obproxy 为每个连接分配了过多的内存，导致内存给撑爆了，触发obproxy 。
麻烦请问修改的参数是？