obproxy进程出现 coredump

OceanBase 技术问题
#1

【 使用环境 】生产环境
【 OB or 其他组件 】obproxy
【 使用版本 】 4.3.2.0-42
【问题描述】进程出现 coredump
【复现路径】偶发，无法复现
【附件及日志】
obproxy宕机前的日志，报错-4257
[2026-04-09 09:12:44.554537] WDIAG [PROXY] get_or_create_metric (ob_prometheus_processor.cpp:455) [347534][Y0-00007F14ACD50AF0] [lt=0] [dc=0] metric num reach limit, will discard and expire metric(metric_num=3000)
[2026-04-09 09:12:44.554572] WDIAG [PROXY] decompress (ob_zlib_stream_compressor.cpp:223) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] fail to decpmress(zlib_ret=-3, stream_.avail_out=8192, ret=-4257)
[2026-04-09 09:12:44.554584] WDIAG [PROXY] handle_analyze_compressed_oceanbase_payload (ob_resp_analyzer.cpp:565) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] fail to decompress(ob20_pkt=0x7ffce3bd4fb0, DEFAULT_PKT_BUFFER_SIZE=8192, filled_len=0, ret=-4257)
[2026-04-09 09:12:44.554587] WDIAG [PROXY] stream_analyze_compressed_oceanbase (ob_resp_analyzer.cpp:1177) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] fail to handle_analyze_compressed_oceanbase_payload(ret=-4257)
[2026-04-09 09:12:44.554588] WDIAG [PROXY] analyze_response (ob_resp_analyzer.cpp:1342) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] fail to stream_analyze_packets(resp_buf=^Y, ret=-4257)
[2026-04-09 09:12:44.554591] WDIAG [PROXY] analyze_all_packets (ob_resp_analyzer.cpp:950) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] fail to analyze_response(ret=-4257)
[2026-04-09 09:12:44.554592] WDIAG [PROXY] analyze_response (ob_resp_analyzer.cpp:1422) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] fail to analyze_all_packets(fail to analyze_all_packets(result={status_:0, mysql_header_:{pkt_len:0, pkt_seq:0, cmd:0, pkt_type:0}, ob20_header_:{ob 2
0 protocol header:{compressed_len:25, seq:1, non_compressed_len:310015}, magic_num:18467, header_checksum:8, connection_id:1850290224, request_id:7102838, pkt_seq:105, payload_len:1918967908, version:12377, flag_.flags:1701672295, reserved:29806}, compressed_mysql_header_:{compress
ed_len:25, seq:1, non_compressed_len:310015}}, resp_result={is_trans_completed:false, is_resp_completed:false, ending_type:7, is_partition_hit:true, has_new_sys_var:false, has_proxy_idc_name_user_var:false, is_server_db_reset:false, reserved_ok_len_of_mysql:0, reserved_ok_len_of_co
mpressed:0, connection_id:0, scramble_buf:"", is_resultset_resp:false, server_capabilities_lower_.capability:0, ok_packet_action_type:0, last_ok_pkt_len:0, rewritten_last_ok_pkt_len:0, extra_info:{is_exist_sess_info:false, sess_info_count:0, extra_len:0}}, ret=-4257)
[2026-04-09 09:12:44.554620] WDIAG [PROXY.SM] handle_first_compress_response_packet (ob_mysql_sm.cpp:4376) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] fail to analyze response(server_buffer_reader_=0x7f14aa723a68, need_receive_completed=false, resp_result={is_trans_completed:false,
is_resp_completed:false, ending_type:7, is_partition_hit:true, has_new_sys_var:false, has_proxy_idc_name_user_var:false, is_server_db_reset:false, reserved_ok_len_of_mysql:0, reserved_ok_len_of_compressed:0, connection_id:0, scramble_buf:"", is_resultset_resp:false, server_capabil
ities_lower_.capability:0, ok_packet_action_type:0, last_ok_pkt_len:0, rewritten_last_ok_pkt_len:0, extra_info:{is_exist_sess_info:false, sess_info_count:0, extra_len:0}}, sm_id=125272675, ret=-4257)
[2026-04-09 09:12:44.554626] WDIAG [PROXY.SM] handle_oceanbase_first_response_packet (ob_mysql_sm.cpp:4325) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] fail to handle_first_compress_response_packet(need_receive_completed=false, ret=-4257)
[2026-04-09 09:12:44.554628] WDIAG [PROXY.SM] handle_first_response_packet (ob_mysql_sm.cpp:4303) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] fail to handle oceanbase first response packet(ret=-4257)
[2026-04-09 09:12:44.554628] WDIAG [PROXY.SM] state_server_response_read (ob_mysql_sm.cpp:3987) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] fail to handle first response packet(ret=-4257)
[2026-04-09 09:12:44.554629] WDIAG [PROXY.SM] state_server_response_read (ob_mysql_sm.cpp:4030) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] Error parsing server response(sm_id=125272675, ret=-4257)
[2026-04-09 09:12:44.554631] WDIAG [PROXY.SM] handle_server_setup_error (ob_mysql_sm.cpp:8987) [347528][Y0-00007F149FD65380] [lt=0] [dc=0] trace_log(trans_state_.trace_log_= | dropped=0)
CRASH ERROR !!! OB_ABORT, tid: 347535, lbt: 0x240933 0x262d2c 0x40fd86 0x7f14b4f71b20 0x7f14b4977f04 0x718add 0xab6bfa 0xaabd69 0x6d9415 0x6e5c1d 0x6f3959 0x6f24aa 0x2982e7 0x29e3e8 0x2816a1 0x14dd70f 0x14d85f1 0x63dcdf 0x7f14b4f6714a 0x7f14b4916f23

coredump目录下生成了core文件，使用gdb打印的堆栈如下：
Core was generated by `/home/admin/obproxy/bin/obproxy -p 2883 -n sdproxy -o server_tcp_keepcnt=2,clie’.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f14b48517ff in raise () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f14a6c4f700 (LWP 347535))]
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-127.el8.x86_64 libblkid-2.32.1-24.el8.x86_64 libgcc-8.3.1-5.1.el8.x86_64 libmount-2.32.1-24.el8.x86_64 libselinux-2.9-3.el8.x86_64 libuuid-2.32.1-24.el8.x86_64 pcre2-10.32-2.el8.x86_64 systemd-libs-239-40.el8.x86_64
(gdb) bt
#0 0x00007f14b48517ff in raise () from /lib64/libc.so.6
#1 0x00007f14b483bc35 in abort () from /lib64/libc.so.6
#2 0x0000562894d2ed6a in ob_abort () at lib/alloc/alloc_assist.cpp:23
#3 0x0000562894edbd86 in oceanbase::obproxy::ObProxyMain::sig_direct_handler (sig=) at obproxy/ob_proxy_main.cpp:950
#4
#5 0x00007f14b4977f04 in __memmove_avx_unaligned_erms () from /lib64/libc.so.6
#6 0x00005628951e4add in oceanbase::obproxy::proxy::ObProxyMysqlRequest::get_sql (this=0x7f149ad82438) at …/src/obproxy/proxy/mysqllib/ob_proxy_mysql_request.h:350
#7 0x0000562895582bfa in oceanbase::obproxy::proxy::ObProxyMysqlRequest::get_parse_sql (this=0x7f149ad82438) at obproxy/proxy/mysqllib/ob_proxy_mysql_request.h:73
#8 oceanbase::obproxy::proxy::ObMysqlRequestAnalyzer::do_analyze_request (ctx=…, sql_cmd=, auth_request=…, client_request=…, is_oracle_mode=is_oracle_mode@entry=false) at obproxy/proxy/mysqllib/ob_mysql_request_analyzer.cpp:621
#9 0x0000562895577d69 in oceanbase::obproxy::proxy::ObMysqlRequestAnalyzer::analyze_request (ctx=…, auth_request=…, client_request=…, sql_cmd=@0x7f149ad85548: oceanbase::obmysql::OB_MYSQL_COM_STMT_PREPARE_EXECUTE,
status=@0x7f14a6c1df6c: oceanbase::obproxy::proxy::ANALYZE_DONE, is_oracle_mode=, is_client_support_ob20_protocol=false) at obproxy/proxy/mysqllib/ob_mysql_request_analyzer.cpp:174
#10 0x00005628951a5415 in oceanbase::obproxy::proxy::ObMysqlSM::analyze_mysql_request (this=this@entry=0x7f149ad820c0, status=@0x7f14a6c1df6c: oceanbase::obproxy::proxy::ANALYZE_DONE, is_mysql_req_in_ob20=is_mysql_req_in_ob20@entry=false)
at …/src/obproxy/proxy/mysqllib/ob_proxy_session_info.h:562
#11 0x00005628951b1c1d in oceanbase::obproxy::proxy::ObMysqlSM::handle_first_request_packet (this=this@entry=0x7f149ad820c0, status=@0x7f14a6c1df6c: oceanbase::obproxy::proxy::ANALYZE_DONE, first_packet_len=@0x7f14a6c1df70: 0) at obproxy/proxy/mysql/ob_mysql_sm.cpp:4705
#12 0x00005628951bf959 in oceanbase::obproxy::proxy::ObMysqlSM::state_client_request_read (this=0x7f149ad820c0, event=, data=) at obproxy/proxy/mysql/ob_mysql_sm.cpp:691
#13 0x00005628951be4aa in oceanbase::obproxy::proxy::ObMysqlSM::main_handler (this=0x7f149ad820c0, event=, data=0x7f149fd5e178) at obproxy/proxy/mysql/ob_mysql_sm.cpp:5630
#14 0x0000562894d642e7 in oceanbase::obproxy::event::ObContinuation::handle_event (data=0x7f149fd5e178, event=, this=) at …/src/obproxy/iocore/eventsystem/ob_continuation.h:106
#15 oceanbase::obproxy::net::ObUnixNetVConnection::read_signal_and_update (this=this@entry=0x7f149fd5df00, event=, event@entry=100) at obproxy/iocore/net/ob_unix_net_vconnection.cpp:113
#16 0x0000562894d6a3e8 in oceanbase::obproxy::net::ObUnixNetVConnection::handle_read_from_net_success (thread=…, total_read=, mutex=0x7f149fd64a40, this=0x7f149fd5df00) at obproxy/iocore/net/ob_unix_net_vconnection.cpp:398
#17 oceanbase::obproxy::net::ObUnixNetVConnection::handle_read_from_net_success (total_read=, mutex=0x7f149fd64a40, thread=…, this=0x7f149fd5df00) at obproxy/iocore/net/ob_unix_net_vconnection.cpp:377
#18 oceanbase::obproxy::net::ObUnixNetVConnection::read_from_net (this=0x7f149fd5df00, thread=…) at obproxy/iocore/net/ob_unix_net_vconnection.cpp:640
#19 0x0000562894d4d6a1 in oceanbase::obproxy::net::ObNetHandler::main_net_event (this=0x5628a7bdc030, event=, e=) at obproxy/iocore/net/ob_unix_net.cpp:654
#20 0x0000562895fa970f in oceanbase::obproxy::event::ObContinuation::handle_event (data=0x7f14acf50cb0, event=5, this=) at …/src/obproxy/iocore/eventsystem/ob_continuation.h:106
#21 oceanbase::obproxy::event::ObEThread::process_event (this=0x5628a7c060a0, e=0x7f14acf50cb0, calling_code=5) at obproxy/iocore/eventsystem/ob_ethread.cpp:300
#22 0x0000562895fa45f1 in oceanbase::obproxy::event::ObEThread::execute (this=) at obproxy/iocore/eventsystem/ob_ethread.cpp:443
#23 0x0000562895109cdf in oceanbase::obproxy::event::spawn_thread_internal (a=0x5628a7c084c0) at obproxy/iocore/eventsystem/ob_thread.cpp:69
#24 0x00007f14b4f6714a in start_thread () from /lib64/libpthread.so.0
#25 0x00007f14b4916f23 in clone () from /lib64/libc.so.6

5 个赞
#3

bin/obproxy --version 具体的版本号发一下
就是突然宕机了是么？完整的obproxy.log obproxy_error.log 日志能提供好一下么？

2 个赞
#4

自动重启了，并产生了coredump文件
REVISION: 1-local-f877a496d4c55b184ec568599ae565616437034d
BUILD_TIME: Dec 6 2024 17:38:33
BUILD_FLAGS: -g -O2 -D_OB_VERSION=1000 -D_NO_EXCEPTION -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DNDEBUG -D__USE_LARGEFILE64 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILE -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -Wall -Wextra -Wunused-parameter -Wformat -Wconversion -Wno-deprecated -
Wno-invalid-offsetof -fno-reorder-blocks-and-partition -finline-functions -fno-strict-aliasing -mtune=core2 -Wno-psabi -Wno-sign-compare -Wno-class-memaccess -Wno-deprecated-copy -Wno-ignored-qualifiers -Wno-aligned-new -Wno-format-truncation -Wno-literal-suffix -Wno-format-overflo
w -Wno-stringop-truncation -Wno-memset-elt-size -Wno-cast-function-type -Wno-address-of-packed-member -fno-omit-frame-pointer -Wl,–emit-relocs,-z,noexecstack,-z,relro,-z,now,-z,notext -fPIC -isystem /home/jenkins/agent/workspace/ob_artifacte_local_artifact/ob_source_code_dir/obpro
xy-tmp.140/BUILD/obproxy-ce-4.3.2.0/deps/3rd/usr/local/oceanbase/deps/devel/include -isystem /home/jenkins/agent/workspace/ob_artifacte_local_artifact/ob_source_code_dir/obproxy-tmp.140/BUILD/obproxy-ce-4.3.2.0/deps/3rd/usr/include -L/home/jenkins/agent/workspace/ob_artifacte_local
_artifact/ob_source_code_dir/obproxy-tmp.140/BUILD/obproxy-ce-4.3.2.0/deps/3rd/usr/local/oceanbase/deps/devel/lib -D_GLIBCXX_USE_CXX11_ABI=0 -DGCC_52 -D_GLIBCXX_USE_CXX11_ABI=0 -DBUILD_OPENSOURCE -DSUPPORT_SSE4_2 -DHAVE_SCHED_GETCPU -DHAVE_REALTIME_COARSE -DOB_HAVE_EVENTFD -DHAVE_F
ALLOCATE -I/home/jenkins/agent/workspace/ob_artifacte_local_artifact/ob_source_code_dir/obproxy-tmp.140/BUILD/obproxy-ce-4.3.2.0/deps/3rd/usr/local/oceanbase/deps/devel/include -I/home/jenkins/agent/workspace/ob_artifacte_local_artifact/ob_source_code_dir/obproxy-tmp.140/BUILD/obpr
oxy-ce-4.3.2.0/deps/3rd/usr/local/oceanbase/deps/devel/include/mariadb -I/home/jenkins/agent/workspace/ob_artifacte_local_artifact/ob_source_code_dir/obproxy-tmp.140/BUILD/obproxy-ce-4.3.2.0/deps/3rd/usr/include -Werror

Copyright (c) 2021 OceanBase
OceanBase Database Proxy(ODP) is licensed under Mulan PubL v2.
You can use this software according to the terms and conditions of the Mulan PubL v2.
You may obtain a copy of Mulan PubL v2 at:
http://license.coscl.org.cn/MulanPubL-2.0
THIS SOFTWARE IS PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OF ANY KIND,
EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
See the Mulan PubL v2 for more details.

2 个赞
#5

你这个查看的版本信息 没有粘贴完吧

2 个赞
#6

ocp上查看的是4.3.2.0-42

3 个赞
#7

in/obproxy --version 的结果也粘贴在上面了的

2 个赞
#8

完整的obproxy.log obproxy_error.log 日志能提供好一下么？

2 个赞
#9

建议升级到436版本 怀疑是触发了问题 目前官网432版本也下掉了

2 个赞
#10

666

2 个赞
#11

好问题，收藏

1 个赞
#12

好的，我们升级解决下

#13

先排除一下操作系统和硬件的问题

#14

学一下