【 使用环境 】、测试环境
【 OB or 其他组件 】
【 使用版本 】
【问题描述】OCP访问突然报500，如何排查与解决
以下为 OCP服务与OCP元数据库信息
4 个赞
发下 ocp-server.log ，默认在 /home/admin/ocp/log
以及access.log，默认在 /home/admin/ocp-server/log
2 个赞
是通过10.1.224.53:2883 这个obproxy连接OCP的元数据集群 ocp_meta租户及ocp_monitor租户的吗？
2025-12-10 09:59:35.131 WARN 11577 --- [alarm-detect-2,1225fb183d11a01f,7b8e7e619f0413f3] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 08000
2025-12-10 09:59:35.131 ERROR 11577 --- [alarm-detect-2,1225fb183d11a01f,7b8e7e619f0413f3] o.h.engine.jdbc.spi.SqlExceptionHelper : monitor-connect-pool - Connection is not available, request timed out after 2000ms.
2025-12-10 09:59:35.131 WARN 11577 --- [alarm-detect-2,1225fb183d11a01f,7b8e7e619f0413f3] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: -1, SQLState: 08000
2025-12-10 09:59:35.131 ERROR 11577 --- [alarm-detect-2,1225fb183d11a01f,7b8e7e619f0413f3] o.h.engine.jdbc.spi.SqlExceptionHelper : Could not connect to 10.1.224.53:2883 : Could not connect to 10.1.224.53:2883 : unexpected end of stream, read 0 bytes from 4
2025-12-10 09:59:35.132 ERROR 11577 --- [alarm-detect-2,1225fb183d11a01f,7b8e7e619f0413f3] c.o.o.a.core.task.AlarmDetectMetricTask : detect failed, alarmType=ob_tenant_long_trans_exist
org.springframework.dao.DataAccessResourceFailureException: Unable to acquire JDBC Connection; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Caused by: java.sql.SQLTransientConnectionException: monitor-connect-pool - Connection is not available, request timed out after 2000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
... 70 common frames omitted
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to 10.1.224.53:2883 : Could not connect to 10.1.224.53:2883 : unexpected end of stream, read 0 bytes from 4
2 个赞
取下包含这个时间 2025-12-10 09:59:35 的10.1.224.53 obproxy.log，obproxy_error.log 以及OCP元数据集群的 observer.log 压缩发下
另外发下obproxy的版本，observer的详细版本
/home/admin/obproxy/bin/obproxy -V
show variables like '%version_comment%';
2 个赞
在线学习等结果看看
1 个赞
obproxy的内存分配较少，内存不足，调大proxy_mem_limited 到6G 再观察下
[2025-12-10 09:59:35.596632] ERROR [PROXY] do_monitor_mem (ob_proxy_main.cpp:1291) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] obproxy's memory is out of limit, will disable alloc memory from the OS(mem_limited=524288000, OTHER_MEMORY_SIZE=73400320, is_out_of_mem_limit=true, cur_pos=3)
[2025-12-10 09:59:35.596648] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1293) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] history memory size, history_mem_size[0]=450887680 BACKTRACE:0x10f16e9 0x10da67d 0x10db03e 0x10d7cc3 0x269a6c7 0x2051220 0x204e4fa 0x1212073 0x12100d3 0x1219c5e 0x7fdd4feb1ea5 0x7fdd503ccb0d
[2025-12-10 09:59:35.596677] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1293) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] history memory size, history_mem_size[1]=450887680 BACKTRACE:0x10f16e9 0x10da67d 0x10db03e 0x10d7cc3 0x269a6c7 0x2051220 0x204e4fa 0x1212073 0x12100d3 0x1219c5e 0x7fdd4feb1ea5 0x7fdd503ccb0d
[2025-12-10 09:59:35.596687] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1293) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] history memory size, history_mem_size[2]=450887680 BACKTRACE:0x10f16e9 0x10da67d 0x10db03e 0x10d7cc3 0x269a6c7 0x2051220 0x204e4fa 0x1212073 0x12100d3 0x1219c5e 0x7fdd4feb1ea5 0x7fdd503ccb0d
[2025-12-10 09:59:35.596697] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1293) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] history memory size, history_mem_size[3]=450887680 BACKTRACE:0x10f16e9 0x10da67d 0x10db03e 0x10d7cc3 0x269a6c7 0x2051220 0x204e4fa 0x1212073 0x12100d3 0x1219c5e 0x7fdd4feb1ea5 0x7fdd503ccb0d
[2025-12-10 09:59:35.596707] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1293) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] history memory size, history_mem_size[4]=450887680 BACKTRACE:0x10f16e9 0x10da67d 0x10db03e 0x10d7cc3 0x269a6c7 0x2051220 0x204e4fa 0x1212073 0x12100d3 0x1219c5e 0x7fdd4feb1ea5 0x7fdd503ccb0d
[2025-12-10 09:59:35.596716] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1293) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] history memory size, history_mem_size[5]=450887680 BACKTRACE:0x10f16e9 0x10da67d 0x10db03e 0x10d7cc3 0x269a6c7 0x2051220 0x204e4fa 0x1212073 0x12100d3 0x1219c5e 0x7fdd4feb1ea5 0x7fdd503ccb0d
[2025-12-10 09:59:35.596727] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1293) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] history memory size, history_mem_size[6]=450887680 BACKTRACE:0x10f16e9 0x10da67d 0x10db03e 0x10d7cc3 0x269a6c7 0x2051220 0x204e4fa 0x1212073 0x12100d3 0x1219c5e 0x7fdd4feb1ea5 0x7fdd503ccb0d
[2025-12-10 09:59:35.596737] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1293) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] history memory size, history_mem_size[7]=450887680 BACKTRACE:0x10f16e9 0x10da67d 0x10db03e 0x10d7cc3 0x269a6c7 0x2051220 0x204e4fa 0x1212073 0x12100d3 0x1219c5e 0x7fdd4feb1ea5 0x7fdd503ccb0d
[2025-12-10 09:59:35.596748] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1293) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] history memory size, history_mem_size[8]=450887680 BACKTRACE:0x10f16e9 0x10da67d 0x10db03e 0x10d7cc3 0x269a6c7 0x2051220 0x204e4fa 0x1212073 0x12100d3 0x1219c5e 0x7fdd4feb1ea5 0x7fdd503ccb0d
[2025-12-10 09:59:35.596758] EDIAG [PROXY] do_monitor_mem (ob_proxy_main.cpp:1293) [26906][Y0-00007FDD432A4FD0] [lt=0] [dc=0] history memory size, history_mem_size[9]=450887680 BACKTRACE:0x10f16e9 0x10da67d 0x10db03e 0x10d7cc3 0x269a6c7 0x2051220 0x204e4fa 0x1212073 0x12100d3 0x1219c5e 0x7fdd4feb1ea5 0x7fdd503ccb0d
怎么查看obproxy的内存分配，以及修改配置大小
黑屏这样操作
obclient -hxxx -P2883 -uroot@proxysys -p'xxx' -Doceanbase -A
show proxyconfig like '%proxy_mem_limited%'\G
alter proxyconfig set proxy_mem_limited=6G;
show proxyconfig like '%proxy_mem_limited%'\G
3 个赞
这种方式重启之后就不生效了。
问题1：obd部署的服务除了obd cluster edit-config 改配置永久生效，还有其他通过命令行的方式永久生效吗？
问题2： OCP 部署的 obporxy 是这样调整配置且永久生效吗
如果obd里面没有配置proxy_mem_limited，黑屏修改后重启也会一直生效
问题1:没有
问题2:是的，永久生效
1 个赞
学习一下，很好的经验
proxy_mem_limited 应该是配置了500M
是的 我obd部署集群的时候配置文件有这个参数
只是给OCP用的，配置2G也可以