连接三副本集群2883端口不定时出现异常 直连副本的2881端口正常

OBServer版本:5.7.25-OceanBase_CE-v4.3.5.3
OBProxy版本:obproxy (OceanBase 4.3.5.0 3.el8)

A、B、C、D四台服务器安装了三副本。其中A服务器安装OBProxy对外2883对业务系统提供服务。

系统是asp.net core,数据访问驱动是Mysqlconnector。不定时访问数据库会出现异常:

MySqlConnector.MySqlException (0x80004005): Failed to read the result set.
 ---> MySqlConnector.MySqlEndOfStreamException: An incomplete response was received from the server
   at MySqlConnector.Protocol.Serialization.ProtocolUtility.ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func`1 getNextSequenceNumber, ArraySegmentHolder`1 previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior) in /_/src/MySqlConnector/Protocol/Serialization/ProtocolUtility.cs:line 424

相关Mysqlconnector的代码如下:

public static async ValueTask<ArraySegment<byte>> ReadPayloadAsync(BufferedByteReader bufferedByteReader, IByteHandler byteHandler, Func<int> getNextSequenceNumber, ArraySegmentHolder<byte> previousPayloads, ProtocolErrorBehavior protocolErrorBehavior, IOBehavior ioBehavior)
{
	previousPayloads.Clear();
	while (true)
	{
		// read the packet header
		var headerBytes = await bufferedByteReader.ReadBytesAsync(byteHandler, 4, ioBehavior).ConfigureAwait(false);
		if (headerBytes.Count < 4)
		{
			return protocolErrorBehavior == ProtocolErrorBehavior.Ignore ? default :
				throw new MySqlEndOfStreamException(4, headerBytes.Count);
		}

Mysqlconnector驱动的代码如何读取服务器响应太过于底层看不懂,只是能知道是这里抛的异常

查阅OBProxy在抛异常时间点的日志如下:
OBProxy日志片段.txt (60.4 KB)

只要通过2883连接,业务系统就会不定时出现异常,但是直连某个Server的2881就没问题,请大神帮忙分析一下怎么回事,感谢

全部日志:
obproxy.log.20251103150503.zip (8.4 MB)

这个日志文件也发一下obproxy_diagnosis.log
bin/obproxy --version 版本信息查看一下

obproxy_diagnosis.log.20251104092313.log.zip (6.3 MB)

bin/obproxy --version

root@ecs-95745332:~/obproxy/bin# ./obproxy --version
./obproxy --version
obproxy (OceanBase 4.3.5.0 3.el8)
REVISION: 1-local-2d7944e0fdf200cd617f558c24c19cf24ef17472
BUILD_TIME: Aug  7 2025 17:01:56
BUILD_FLAGS: -g -O2 -D_OB_VERSION=1000 -D_NO_EXCEPTION -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DNDEBUG -D__USE_LARGEFILE64 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILE -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -Wformat -finline-functions -fno-strict-aliasing -mtune=core2 -fno-omit-frame-pointer -Wl,-z,noexecstack,-z,relro,-z,now,-z,notext -fPIC -isystem /home/jenkins/agent/workspace/ob_artifacte_local_artifact/ob_source_code_dir/obproxy-tmp.158/BUILD/obproxy-ce-4.3.5.0/deps/3rd/usr/local/oceanbase/deps/devel/include -isystem /home/jenkins/agent/workspace/ob_artifacte_local_artifact/ob_source_code_dir/obproxy-tmp.158/BUILD/obproxy-ce-4.3.5.0/deps/3rd/usr/include -L/home/jenkins/agent/workspace/ob_artifacte_local_artifact/ob_source_code_dir/obproxy-tmp.158/BUILD/obproxy-ce-4.3.5.0/deps/3rd/usr/local/oceanbase/deps/devel/lib -D_GLIBCXX_USE_CXX11_ABI=0 -DBUILD_OPENSOURCE -DSUPPORT_SSE4_2 -DHAVE_SCHED_GETCPU -DHAVE_REALTIME_COARSE -DOB_HAVE_EVENTFD -DHAVE_FALLOCATE -fuse-ld=lld -Wall -Wextra -Wno-invalid-offsetof -Wno-address-of-packed-member -Wno-deprecated -Wno-psabi -Wno-ignored-qualifiers -Wno-deprecated-copy -Wno-sign-compare -Wno-varargs -Wno-overloaded-virtual -Wno-sign-conversion -Wno-string-plus-int -Wno-shorten-64-to-32 -Wno-delete-non-abstract-non-virtual-dtor -Wno-unused-command-line-argument -Wno-inconsistent-missing-override -Wno-mismatched-tags -Wno-dynamic-class-memaccess -Wno-format-security -Wno-reinterpret-base-class -Wl,--allow-multiple-definition --gcc-toolchain=/home/jenkins/agent/workspace/ob_artifacte_local_artifact/ob_source_code_dir/obproxy-tmp.158/BUILD/obproxy-ce-4.3.5.0/deps/3rd/usr/local/oceanbase/devtools

Copyright (c) 2021 OceanBase
OceanBase Database Proxy(ODP) is licensed under Mulan PubL v2.
You can use this software according to the terms and conditions of the Mulan PubL v2.
You may obtain a copy of Mulan PubL v2 at:
         http://license.coscl.org.cn/MulanPubL-2.0
THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
See the Mulan PubL v2 for more details.

连接的串的信息也发一下吧

Server=192.168.0.242;Port=2883;Database=guangming;charset=utf8;user=root@mytenant;password=xxx;Allow User Variables=true;SslMode=none

你连接obproxy 查询一下 show proxyconfig like ‘%enable_ob_protocol_v2_with_client%’;

obclient(root@proxysys@proxysys)[oceanbase]>  show proxyconfig like '%enable_ob_protocol_v2_with_client%';
+-----------------------------------+-------+---------------------------------------------------------------+-------------+---------------+-------+--------------+
| name                              | value | info                                                          | need_reboot | visible_level | range | config_level |
+-----------------------------------+-------+---------------------------------------------------------------+-------------+---------------+-------+--------------+
| enable_ob_protocol_v2_with_client | False | if enabled, proxy will use oceanbase protocol 2.0 with client | false       | USER          |       | LEVEL_GLOBAL |
+-----------------------------------+-------+---------------------------------------------------------------+-------------+---------------+-------+--------------+
1 row in set (0.001 sec)

这个日志也提供一下 obproxy_error.log

obproxy_error.zip (4.5 MB)

2025-11-03 14:22:49.934980] INFO [PROXY.SS] do_io_close (ob_mysql_server_session.cpp:188) [891543][Y0-000074746C02E3A0] [lt=0] [dc=0] server session do_io_close(server session do_io_close(this=0x7474697ac0f0, ss_id=960811, server_sessid=3222267269, server_addr={192.168.0.193:2881}, server_vc=0x7474695ca7e0, ob_server_cap=11123527, client_session={this:0x74746b86e5f0, session_states_.is_proxy_mysql_client:0, session_states_.is_waiting_trans_first_request:0,
看着像是observer主动断连了 把对应的时间段的192.168.0.193的observer.log日志发一下吧

日志刷新太快昨天的都刷没了。等晚上我把日志都清理一下重现问题后再发吧,辛苦明天看一下

也可以在obproxy连接到observer抓个包 到时候一起发一下

抓包这个就不太懂了,有没有具体的操作过程呢。

wireshark这个工具有用过没有 可以使用这个工具抓包或者使用tcpdump这个工具 在obproxy侧抓个包

158observer.tar.gz (3.3 MB)
193observer.tar.gz (3.9 MB)
221observer.tar.gz (3.3 MB)
obproxy.tar.gz (273.8 KB)
out.zip (10.0 MB)

麻烦看一下。谢谢

ob详细的版本也发一下
SHOW VARIABLES like ‘version_comment’;

OceanBase_CE 4.3.5.3 (r103010012025090210-8b80b225c2dcba7dd0c83f3d5a24e3c1ffc03f24) (Built Sep 2 2025 10:24:21)