ob租户Binlog 服务状态异常,最终导致连接数满了,业务连接不上数据库

【 使用环境 】准生产环境
【 OB or 其他组件 】OceanBase Binlog 服务
【 使用版本 】4.2.5-100000042025052216
【问题描述】obproxy连接数暴涨,初步排查是因为ob租户Binlog 服务状态异常,咱们自研的数据同步工具不断重连obproxy,最终导致连接数满了,业务连接不上数据库
【复现路径】
【附件及日志】

logproxy.log中的error日志:

[2025-09-10 10:22:07.723162] [error] ob_mysql_packet.cpp(81): Failed to receive mysql packet because of poll.revents & POLLERR
[2025-09-10 10:22:07.723202] [error] mysql_protocol.cpp(83): Failed to receive handshake packet from: ./run/cv74u785br/binlog_instance.socket, error: 11(Resource temporarily unavailable)
[2025-09-10 10:22:07.723207] [error] instance_client.cpp(62): Failed to login binlog instance, is_local: true, addr: ./run/cv74u785br/binlog_instance.socket, exceed uds len limit: false
[2025-09-10 10:22:07.723211] [error] instance_client.cpp(42): Failed to connect to instance: cv74u785br, server addr: ./run/cv74u785br/binlog_instance.socket
[2025-09-10 10:22:07.723213] [error] database_protocol.cpp(3688): Failed to detect binlog instance: cv74u785br
[2025-09-10 10:22:07.723223] [error] ob_mysql_packet.cpp(140): Failed to send packet, error: 32(Broken pipe)
[2025-09-10 10:22:07.723225] [error] mysql_protocol.cpp(224): Failed to send query packet to server: ./run/cv74u785br/binlog_instance.socket
[2025-09-10 10:22:07.723226] [error] database_protocol.cpp(3433): Failed to show binlog dumplist for instance:cv74u785br
[2025-09-10 10:22:08.661551] [error] io.cpp(188): Failed to connect to server: ./run/cv74u785br/binlog_instance.socket(server_sun.sun_path: ./run/cv74u785br/binlog_instance.socket), error: 111(Connection refused)
[2025-09-10 10:22:08.661580] [error] mysql_protocol.cpp(66): Failed to connect to server: ./run/cv74u785br/binlog_instance.socket
[2025-09-10 10:22:08.661583] [error] instance_client.cpp(62): Failed to login binlog instance, is_local: true, addr: ./run/cv74u785br/binlog_instance.socket, exceed uds len limit: false
[2025-09-10 10:22:08.661585] [error] instance_client.cpp(42): Failed to connect to instance: cv74u785br, server addr: ./run/cv74u785br/binlog_instance.socket
[2025-09-10 10:22:08.661587] [error] database_protocol.cpp(3688): Failed to detect binlog instance: cv74u785br
[2025-09-10 10:22:08.661593] [error] ob_mysql_packet.cpp(140): Failed to send packet, error: 9(Bad file descriptor)
[2025-09-10 10:22:08.661594] [error] mysql_protocol.cpp(224): Failed to send query packet to server: ./run/cv74u785br/binlog_instance.socket
[2025-09-10 10:22:08.661596] [error] database_protocol.cpp(3433): Failed to show binlog dumplist for instance:cv74u785br


binlog4.2.5 不支持ob435bp3

1 个赞

咱们有多个租户,有些租户binlog服务是正常的,就这一个租户的binlog服务有问题呢

请问上线多长时间了呢?

设置一下允许用户的最大连接数吧

集群是刚升级到OceanBase_CE 4.3.5.3版本,binlog服务使用有一段时间了

租户的binlog服务异常后,同步工具会一直重连,客户端的连接会一直涨,暂时把binlog服务异常的租户同步都停了,连接数恢复正常了

估计要热更binlog4.2.5 cdc版本了 要和ob435bp3的cdc版本对齐 要不然会有问题的

1 个赞

是的,咱们先升级了obcdc版本,binlog服务正常了。后续再升级下obbinlog,感谢支持。

当时的logproxy.log日志提供一下 binlog实例出问题的那个实例的binlog_instance.log
和libobcdc.log发一下 我们在看一下