4.4.2.1版本，出现crash error异常，集群中2台机器接连宕机

OceanBase 技术问题
,
#1

【 使用环境 】生产环境
【 OB or 其他组件 】OB
【 使用版本 】4.4.2.1
【问题描述】
宕机前分别告警：
alarm_template_id=0:ob_cluster=oa-1749632252:host=192.168.2.7 OBServer crash

[OBServer crash] 集群：oa，主机：192.168.2.7，日志类型：observer，日志文件：/home/admin/oceanbase/log/observer.log，日志级别：INFO，关键字=CRASH ERROR!!!，错误码=-1，日志详情=[2026-08-20 16:02:30.230947] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:244) [1554041][sql_nio19][T0][Y0-0000000000000000-0-0] [lt=2] connection close(sessid=3224015923, proxy_sessid=13882532270319034335, tenant_id=1006, from_proxy=true, from_java_client=false, c/s protocol=“OB_2_0_CS_TYPE”, is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=YB42C0A80207-0006565BE75E5893-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:1867, cur_pkt_pos_:1873, pkt_rec[1841]:{send:obp_compess_header_{com_len:8687, com_seq:1}, obp20_header_:{payload_len:8659, pkt_seq:1, req_id:13002243, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:5, seq_:2}, pkt_name:“PKT_RESHEAD”, obp_mysql_header_.is_send_:1}, pkt_rec[1842]:{send:obp_compess_header_{com_len:8687, com_seq:1}, obp20_header_:{payload_len:8659, pkt_seq:1, req_id:13002243, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:107, seq_:97}, pkt_name:“PKT_FIELD”, obp_mysql_header_.is_send_:1}, pkt_rec[1843]:{send:obp_compess_header_{com_len:8687, com_seq:1}, obp20_header_:{payload_len:8659, pkt_seq:1, req_id:13002243, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:98}, pkt_name:“PKT_EOF”, obp_mysql_header_.is_send_:1}, pkt_rec[1844]:{send:obp_compess_header_{com_len:8687, com_seq:1}, obp20_header_:{payload_len:8659, pkt_seq:1, req_id:13002243, flag_.flags:10, ext_flags:0}, obp_mysql_header_{pkt_num_:6, seq_:99}, pkt_name:“PKT_ROW”, obp_mysql_header_.is_send_:1}, pkt_rec[1845]:{send:obp_compess_header_{com_len:8687, com_seq:1}, obp20_header_:{payload_len:8659, pkt_seq:1, req_id:13002243, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:9, seq_:100}, pkt_name:“PKT_EOF”, obp_mysql_header_.is_send_:1}, pkt_rec[1846]:{send:obp_compess_header_{com_len:8687, com_seq:1}, obp20_header_:{payload_len:8659, pkt_seq:1, req_id:13002243, flag_.flags:10, ext_flags:0}, obp_mysql_header_{len_:12, seq_:101}, pkt_name:“PKT_OKP”, obp_mysql_header_.is_send_:1}, pkt_rec[1847]:{receive:obp20_header_:{payload_len:107, pkt_seq:0, req_id:13002244, flag_.flags:62,…。

alarm_template_id=0:ob_cluster=oa-1749632252:host=192.168.2.10 OBServer crash

[OBServer crash] 集群：oa，主机：192.168.2.10，日志类型：observer，日志文件：/home/admin/oceanbase/log/observer.log，日志级别：INFO，关键字=CRASH ERROR!!!，错误码=-1，日志详情=[2026-08-20 16:01:18.193822] INFO [RPC.OBMYSQL] sm_conn_unlock_tenant (obsm_conn_callback.cpp:155) [239868][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] unlock session of tenant(conn.sessid_=3222439164, proxy_sessid=13882532270319030672, conn.tenant_id_=1006)CRASH ERROR!!! IP=7fb49bf85f40, RBP=7f04b2361e40, sig=11, sig_code=128, sig_addr=0x0, RLIMIT_CORE=0, timestamp=1787212878186084, tid=3461923, tname=T1006_L0_G0, trace_id=YB42C0A8020A-000655F6EA17AA51-0-0, lbt=0x236cc018 0x2369de31 0x7fb49be5ffef 0x7fb49bf85f40 0x9a7d977 0x13d36839 0x12660e1d 0x1253b9dc 0x1253b422 0x155ab0c1 0x155b0d80 0x1fc0a53a 0x2138ba99 0x92741e5 0x13f79384 0x13f7446b 0x90e461b 0x9519714 0x90e461b 0x13f52db9 0x13f4d542 0x90e461b 0x90e4b42 0x90dfbbc 0x90dcbe3 0x90daa00 0x90b6f2a 0x90ac6eb 0x90a77c3 0x90a53ce 0x909c632 0x1158e4b1 0x22e137fa 0x7fb49beabf8d 0x7fb49bf2b0eb, SQL_ID=, SQL_STRING=WITH recent AS (SELECT plat_order_id, plat_origin_data FROM plat_order_origin WHERE id >= 210133942 AND plat_code = ‘aliexpress’), child_attrs AS (SELECT r.plat_order_id, JSON_UNQUOTE(JSON_EXTRACT(JSON_UNQUOTE(j.extend_map), ‘$.fft_aplLogisticsAttr’)) AS attr FROM recent r, JSON_TABLE(r.plat_origin_data, '$.aliexpress_trade_new_redefining_findorderbyid_response.target.child_order_list.aeop_tp_child_order_dto[]’ COLUMNS (extend_map JSON PATH ‘$.extend_map’)) j WHERE JSON_UNQUOTE(JSON_EXTRACT(JSON_UNQUOTE(j.extend_map), ‘$.fft_isApl’)) = ‘1’), order_attrs AS (SELECT plat_order_id, GROUP_CONCAT(DISTINCT COALESCE(NULLIF(attr, ‘’), ‘[EMPTY]’) ORDER BY attr SEPARATOR ‘+’) AS attr_combo, COUNT() AS apl_child_count FROM child_attrs GROUP BY plat_order_id) SELECT attr_combo, COUNT(*) AS order_count, SUM(apl_child_count) AS apl_child_count, SUM(CASE WHEN INSTR(attr_combo, ‘+’) > 0 THEN 1 ELSE 0 END) AS mixed_order_count FROM order_attrs GROUP BY attr_combo ORDER BY order_count DESC。

#3

把CRASH ERROR!!!前后的observer.log 日志都发一下 这两个节点的都发一下 如果其他的也有 也发一下 rootservice.log也发一下

收集一下这个信息 看看
addr2line -pCfe bin/observer 0x236cc018 0x2369de31 0x7fb49be5ffef 0x7fb49bf85f40 0x9a7d977 0x13d36839 0x12660e1d 0x1253b9dc 0x1253b422 0x155ab0c1 0x155b0d80 0x1fc0a53a 0x2138ba99 0x92741e5 0x13f79384 0x13f7446b 0x90e461b 0x9519714 0x90e461b 0x13f52db9 0x13f4d542 0x90e461b 0x90e4b42 0x90dfbbc 0x90dcbe3 0x90daa00 0x90b6f2a 0x90ac6eb 0x90a77c3 0x90a53ce 0x909c632 0x1158e4b1 0x22e137fa 0x7fb49beabf8d 0x7fb49bf2b0eb

crash 后看看有没有core 文件生成
sysctl -a | grep pattern

kernel.core_pattern = /data/core-%e-%p-%t

SHOW VARIABLES like ‘version_comment’; 版本信息确定一下

如果上面查不出来 下面的命令查一下
bin/observer --version

#4

机器1发生crash日志的observer.log 和rootserver.log
mx7.zip (33.1 MB)

机器2发生crash日志的observer.log 和rootserver.log
ep10.7z (9.9 MB)

没有core文件

版本号
version_comment OceanBase_CE 4.4.2.1 (r101000022026050611-8cf64ed50606966fd5c29f47265cf557d97ea776) (Built May 6 2026 12:17:58)

#5

这个执行一下 收集堆栈信息

#6

safe_backtrace at ??:?
oceanbase::common::coredump_cb(int, int, void*, void*) at ??:?
?? ??:0
?? ??:0
oceanbase::common::ObDiscreteBase::to_rows(oceanbase::sql::RowMeta const&, oceanbase::sql::ObCompactRow**, unsigned short const*, long, long) const at ??:?
oceanbase::sql::ObTempRowStoreBase::add_batch(oceanbase::common::ObIArrayoceanbase::common::ObIVector* const&, unsigned short const*, long, oceanbase::sql::ObCompactRow**, oceanbase::common::ObIArray const*) at ??:?
oceanbase::sql::ObExtendHashTableVec<oceanbase::sql::ObHashPartBucketGeneraloceanbase::sql::ObHashPartItem >::inner_process_batch(oceanbase::sql::RowMeta const&, long, oceanbase::sql::ObBitVectorImpl const*, oceanbase::sql::ObBitVectorImpl&, unsigned long*, std::function<int (oceanbase::common::ObIArrayoceanbase::common::ObIVector* const&, unsigned short const*, long, oceanbase::sql::ObCompactRow**)>, bool, long, long&) at 1_cxx.cxx:?
oceanbase::sql::ObExtendHashTableVec<oceanbase::sql::ObHashPartBucketGeneraloceanbase::sql::ObHashPartItem >::set_distinct_batch(oceanbase::sql::RowMeta const&, long, oceanbase::sql::ObBitVectorImpl const*, oceanbase::sql::ObBitVectorImpl&, unsigned long*, std::function<int (oceanbase::common::ObIArrayoceanbase::common::ObIVector* const&, unsigned short const*, long, oceanbase::sql::ObCompactRow**)>) at 1_cxx.cxx:?
oceanbase::sql::ObHashPartInfrastructureVec<oceanbase::sql::ObHashPartBucketGeneraloceanbase::sql::ObHashPartItem >::set_distinct_batch(oceanbase::common::ObIArrayoceanbase::sql::ObExpr* const&, unsigned long*, long, oceanbase::sql::ObBitVectorImpl const*, oceanbase::sql::ObBitVectorImpl&) at ??:?
oceanbase::sql::ObIHashPartInfrastructure::do_insert_batch_with_unique_hash_table(oceanbase::common::ObIArrayoceanbase::sql::ObExpr* const&, unsigned long*, long, oceanbase::sql::ObBitVectorImpl const*, oceanbase::sql::ObBitVectorImpl&) at ??:?
oceanbase::sql::ObHashPartInfrastructureVecImpl::insert_row_for_batch(oceanbase::common::ObIArrayoceanbase::sql::ObExpr* const&, unsigned long, long, oceanbase::sql::ObBitVectorImpl const*, oceanbase::sql::ObBitVectorImpl&) at ??:?
oceanbase::share::aggregate::HashBasedDistinctVecExtraResult::insert_row_for_batch(oceanbase::common::ObIArrayoceanbase::sql::ObExpr* const&, long, oceanbase::sql::ObBitVectorImpl const, long) at ??:?
oceanbase::share::aggregate::DistinctWrapper<oceanbase::share::aggregate::GroupStoreWrapper<oceanbase::share::aggregate::GroupConcatAggregate<(oceanbase::common::VecValueTypeClass)13> > >::add_batch_rows(oceanbase::share::aggregate::RuntimeContext&, int, oceanbase::sql::ObBitVectorImpl const&, oceanbase::sql::EvalBound const&, char*, oceanbase::share::aggregate::RowSelector) at group_concat.cpp:?
oceanbase::share::aggregate::Processor::add_batch_rows(int, int, char*, oceanbase::sql::ObBatchRows const&, unsigned short, unsigned short) at ??:?
oceanbase::sql::ObMergeGroupByVecOp::aggregate_group_rows(oceanbase::sql::ObBatchRows const&, int, int) at 0_cxx.cxx:?
oceanbase::sql::ObMergeGroupByVecOp::inner_get_next_batch(long) at ??:?
oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) at ??:?
oceanbase::sql::ObSubPlanScanOp::inner_get_next_batch(long) at 1_cxx.cxx:?
oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) at ??:?
oceanbase::sql::ObHashGroupByVecOp::load_data_batch(long) at 0_cxx.cxx:?
oceanbase::sql::ObHashGroupByVecOp::inner_get_next_batch(long) at ??:?
oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) at ??:?
oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) at ??:?
oceanbase::sql::ObExecuteResult::get_next_row(oceanbase::sql::ObExecContext&, oceanbase::common::ObNewRow const*&) at ??:?
oceanbase::observer::ObQueryDriver::response_query_result(oceanbase::sql::ObResultSet&, bool, bool, bool&, long) at 0_cxx.cxx:?
oceanbase::observer::ObSyncPlanDriver::response_result(oceanbase::observer::ObMySQLResultSet&) at 0_cxx.cxx:?
oceanbase::observer::ObMPQuery::process_single_stmt(oceanbase::sql::ObMultiStmtItem const&, oceanbase::observer::ObSMConnection*, oceanbase::sql::ObSQLSessionInfo&, bool, bool, bool&, bool&) at 0_cxx.cxx:?
oceanbase::observer::ObMPQuery::process() at ??:?
oceanbase::rpc::frame::ObSqlProcessor::run() at ??:?
oceanbase::omt::ObWorkerProcessor::process(oceanbase::rpc::ObRequest&) at 0_cxx.cxx:?
oceanbase::omt::ObThWorker::worker(long&, long&, int&) at 0_cxx.cxx:?
non-virtual thunk to oceanbase::omt::ObThWorker::run(long) at 0_cxx.cxx:?
oceanbase::lib::thread::__th_start(void*) at 0_cxx.cxx:?
?? ??:0
?? ??:0

#7

.7机器收集信息：
safe_backtrace at ??:?
oceanbase::common::coredump_cb(int, int, void*, void*) at ??:?
?? ??:0
?? ??:0
oceanbase::common::ObDiscreteBase::to_rows(oceanbase::sql::RowMeta const&, oceanbase::sql::ObCompactRow**, unsigned short const*, long, long) const at ??:?
oceanbase::sql::ObTempRowStoreBase::add_batch(oceanbase::common::ObIArrayoceanbase::common::ObIVector* const&, unsigned short const*, long, oceanbase::sql::ObCompactRow**, oceanbase::common::ObIArray const*) at ??:?
oceanbase::sql::ObExtendHashTableVec<oceanbase::sql::ObHashPartBucketGeneraloceanbase::sql::ObHashPartItem >::inner_process_batch(oceanbase::sql::RowMeta const&, long, oceanbase::sql::ObBitVectorImpl const*, oceanbase::sql::ObBitVectorImpl&, unsigned long*, std::function<int (oceanbase::common::ObIArrayoceanbase::common::ObIVector* const&, unsigned short const*, long, oceanbase::sql::ObCompactRow**)>, bool, long, long&) at 1_cxx.cxx:?
oceanbase::sql::ObExtendHashTableVec<oceanbase::sql::ObHashPartBucketGeneraloceanbase::sql::ObHashPartItem >::set_distinct_batch(oceanbase::sql::RowMeta const&, long, oceanbase::sql::ObBitVectorImpl const*, oceanbase::sql::ObBitVectorImpl&, unsigned long*, std::function<int (oceanbase::common::ObIArrayoceanbase::common::ObIVector* const&, unsigned short const*, long, oceanbase::sql::ObCompactRow**)>) at 1_cxx.cxx:?
oceanbase::sql::ObHashPartInfrastructureVec<oceanbase::sql::ObHashPartBucketGeneraloceanbase::sql::ObHashPartItem >::set_distinct_batch(oceanbase::common::ObIArrayoceanbase::sql::ObExpr* const&, unsigned long*, long, oceanbase::sql::ObBitVectorImpl const*, oceanbase::sql::ObBitVectorImpl&) at ??:?
oceanbase::sql::ObIHashPartInfrastructure::do_insert_batch_with_unique_hash_table(oceanbase::common::ObIArrayoceanbase::sql::ObExpr* const&, unsigned long*, long, oceanbase::sql::ObBitVectorImpl const*, oceanbase::sql::ObBitVectorImpl&) at ??:?
oceanbase::sql::ObHashPartInfrastructureVecImpl::insert_row_for_batch(oceanbase::common::ObIArrayoceanbase::sql::ObExpr* const&, unsigned long, long, oceanbase::sql::ObBitVectorImpl const*, oceanbase::sql::ObBitVectorImpl&) at ??:?
oceanbase::share::aggregate::HashBasedDistinctVecExtraResult::insert_row_for_batch(oceanbase::common::ObIArrayoceanbase::sql::ObExpr* const&, long, oceanbase::sql::ObBitVectorImpl const, long) at ??:?
oceanbase::share::aggregate::DistinctWrapper<oceanbase::share::aggregate::GroupStoreWrapper<oceanbase::share::aggregate::GroupConcatAggregate<(oceanbase::common::VecValueTypeClass)13> > >::add_batch_rows(oceanbase::share::aggregate::RuntimeContext&, int, oceanbase::sql::ObBitVectorImpl const&, oceanbase::sql::EvalBound const&, char*, oceanbase::share::aggregate::RowSelector) at group_concat.cpp:?
oceanbase::share::aggregate::Processor::add_batch_rows(int, int, char*, oceanbase::sql::ObBatchRows const&, unsigned short, unsigned short) at ??:?
oceanbase::sql::ObMergeGroupByVecOp::aggregate_group_rows(oceanbase::sql::ObBatchRows const&, int, int) at 0_cxx.cxx:?
oceanbase::sql::ObMergeGroupByVecOp::inner_get_next_batch(long) at ??:?
oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) at ??:?
oceanbase::sql::ObSubPlanScanOp::inner_get_next_batch(long) at 1_cxx.cxx:?
oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) at ??:?
oceanbase::sql::ObHashGroupByVecOp::load_data_batch(long) at 0_cxx.cxx:?
oceanbase::sql::ObHashGroupByVecOp::inner_get_next_batch(long) at ??:?
oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) at ??:?
oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) at ??:?
oceanbase::sql::ObExecuteResult::get_next_row(oceanbase::sql::ObExecContext&, oceanbase::common::ObNewRow const*&) at ??:?
oceanbase::observer::ObQueryDriver::response_query_result(oceanbase::sql::ObResultSet&, bool, bool, bool&, long) at 0_cxx.cxx:?
oceanbase::observer::ObSyncPlanDriver::response_result(oceanbase::observer::ObMySQLResultSet&) at 0_cxx.cxx:?
oceanbase::observer::ObMPQuery::process_single_stmt(oceanbase::sql::ObMultiStmtItem const&, oceanbase::observer::ObSMConnection*, oceanbase::sql::ObSQLSessionInfo&, bool, bool, bool&, bool&) at 0_cxx.cxx:?
oceanbase::observer::ObMPQuery::process() at ??:?
oceanbase::rpc::frame::ObSqlProcessor::run() at ??:?
oceanbase::omt::ObWorkerProcessor::process(oceanbase::rpc::ObRequest&) at 0_cxx.cxx:?
oceanbase::omt::ObThWorker::worker(long&, long&, int&) at 0_cxx.cxx:?
non-virtual thunk to oceanbase::omt::ObThWorker::run(long) at 0_cxx.cxx:?
oceanbase::lib::thread::__th_start(void*) at 0_cxx.cxx:?
?? ??:0
?? ??:0

#8

能保存到文件里么？有些字符都转译了

#9

告警2.txt (2.1 KB)
告警1.txt (2.3 KB)

#10

你这个怎么触发的 是执行这个语句以后就CRASH ERROR!!!了么
WITH recent AS (SELECT plat_order_id, plat_origin_data FROM plat_order_origin WHERE id >= 210133942 AND plat_code = ‘aliexpress’), child_attrs AS (SELECT r.plat_order_id, JSON_UNQUOTE(JSON_EXTRACT(JSON_UNQUOTE(j.extend_map), ‘$.fft_aplLogisticsAttr’)) AS attr FROM recent r, JSON_TABLE(r.plat_origin_data, '$.aliexpress_trade_new_redefining_findorderbyid_response.target.child_order_list.aeop_tp_child_order_dto[]’ COLUMNS (extend_map JSON PATH ‘$.extend_map’)) j WHERE JSON_UNQUOTE(JSON_EXTRACT(JSON_UNQUOTE(j.extend_map), ‘$.fft_isApl’)) = ‘1’), order_attrs AS (SELECT plat_order_id, GROUP_CONCAT(DISTINCT COALESCE(NULLIF(attr, ‘’), ‘[EMPTY]’) ORDER BY attr SEPARATOR ‘+’) AS attr_combo, COUNT( ) AS apl_child_count FROM child_attrs GROUP BY plat_order_id) SELECT attr_combo, COUNT(*) AS order_count, SUM(apl_child_count) AS apl_child_count, SUM(CASE WHEN INSTR(attr_combo, ‘+’) > 0 THEN 1 ELSE 0 END) AS mixed_order_count FROM order_attrs GROUP BY attr_combo ORDER BY order_count DESC。

#11

image
image1286×418 56.1 KB

大概率是。 cpu飙升然后到宕机。发生很快

#12

从日志看 应该是的
_enable_rich_format 你查一下 这个变量 看看配置是true么？

#13

从哪看。 租户和集群都没看到这个参数

#14

是个变量 show variables like 这样查看 命令行查看

#15

image
image943×552 7.78 KB

image
image911×219 4.12 KB

没有结果

#16

select
b.tenant_name,
a.tenant_id,
tenant_type,
name,
value,
info,
scope
from
CDB_OB_SYS_VARIABLES a,
DBA_OB_TENANTS b
where
a.tenant_id = b.tenant_id
and name like ‘%_nlj_batching_enabled%’
and tenant_type not in (‘SYS’,‘META’); 这样查一下

#17
tenant_name tenant_id tenant_type name value info scope
epean 1006 USER _nlj_batching_enabled 1 enable batching of the RHS IO in NLJ GLOBAL
#18

SET SESSION _enable_rich_format = false; select @@_enable_rich_format; 在查一下 你临时关闭 在跑一下 看看是否还会CRASH ERROR!!!

#19

image

没有这个参数

#20

设置不生效么SET SESSION _enable_rich_format = false; 这样 设置报错么？还是查询报错

#21

image
image1231×116 4.55 KB

设置报错