死锁问题,集群不可用

[2026-08-09 11:40:45.863016] INFO [STORAGE] reset (ob_tablet.cpp:243) [7454][TimerWK1_BlkMgr][T1002][Y0-0000000000000000-0-0] [lt=22] reset tablet(this=0x14769f1b4060, ls_id={id:1003}, tablet_id={id:1152921504606895625}, lbt()=“0xa916688 0x1d6f3aa3 0x1d6f31c5 0x1d6f2773 0xa4cd87e 0xa8d5f70 0xaccca7f 0x1ce3f9d2 0xa7b3fa8 0xa7a300d 0x260bc515 0x260ba77e 0x147799ab6a83 0x147799b48890”)
[2026-08-09 11:40:45.863040] INFO [STORAGE] reset (ob_tablet.cpp:243) [7454][TimerWK1_BlkMgr][T1002][Y0-0000000000000000-0-0] [lt=7] reset tablet(this=0x14769f1b4060, ls_id={id:-1}, tablet_id={id:0}, lbt()=“0xa916688 0x1d6f3aa3 0x1d6f31c5 0xa8301fe 0xa8d6435 0xa8d5f8b 0xaccca7f 0x1ce3f9d2 0xa7b3fa8 0xa7a300d 0x260bc515 0x260ba77e 0x147799ab6a83 0x147799b48890”)
[2026-08-09 11:40:45.863057] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9775][T1002_L0_G0][T1002][YB42AC175339-00065894F9FA03C4-0-0] [lt=1] mvcc_write conflict(key={“INT”:4}, tx_id={txid:20000905}, conflict_tx_id={txid:20000866}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1786177034366336 max_table_version=1786177034366336 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246832022405 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20000905} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=9555, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(9556,1,0,0,9556,9555,9555,0)|]}})
[2026-08-09 11:40:45.863213] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9835][T1002_L0_G0][T1002][YB42AC175339-00065894F9EA0367-0-0] [lt=0] mvcc_write conflict(key={“INT”:4}, tx_id={txid:20001052}, conflict_tx_id={txid:20000866}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1786177034366336 max_table_version=1786177034366336 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246845132705 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20001052} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=400, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(401,1,0,0,401,400,400,0)|]}})
[2026-08-09 11:40:45.863346] INFO [STORAGE.TRANS] rollback_savepoint_slowpath_ (ob_tx_api.cpp:1832) [9291][T1002_L0_G0][T1002][YB42AC175339-00065894F9FA02DC-0-0] [lt=1] rollback savepoint slowpath(ret=0, tx.tx_id={txid:20000807}, start_ts=1786246845862197, retries=0, savepoint={branch:0, seq:192827257}, expire_ts=1786247678474765, tx={this:0x1476ba661ae0, tx_id:{txid:20000807}, state:2, addr:“172.23.83.57:2882”, tenant_id:1002, session_id:3221510608, assoc_session_id:3221510608, client_sid:3221510608, xid:NULL, xa_mode:"", xa_start_addr:“0.0.0.0:0”, access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:{branch:0, seq:1}, op_sn:9005, alloc_ts:1786246652992668, active_ts:1786246652992668, commit_ts:-1, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1786333052992668, coord_id:{id:-1}, parts:[{id:{id:1002}, addr:“172.28.253.212:2882”, epoch:3384236081325798, first_scn:{branch:0, seq:85575482}, last_scn:MAX, last_touch_ts:1786246738570356, flag_.flag_bit_:{is_dup_ls:0, is_clean:false}}], exec_info_reap_ts:1786246738570355, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1720306770, cluster_version:17180067078, seq_base:1786246652991728, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:1786246845819857000, v:0}, ref:2}, rollback_parts.count()=1)
[2026-08-09 11:40:45.863516] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9814][T1002_L0_G0][T1002][YB42AC175339-00065894F81A0423-0-0] [lt=1] mvcc_write conflict(key={“INT”:16}, tx_id={txid:20000907}, conflict_tx_id={txid:20000940}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1786177034366336 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246840091118 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20000907} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=4470, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(4471,1,0,0,4471,4470,4470,0)|]}})
[2026-08-09 11:40:45.863617] INFO [STORAGE] inc_ref_with_macro_iter (ob_tablet.cpp:3400) [8289][T1002_TimerWK6_WriteCkpt][T1002][YB42AC175339-00065894F649FD85-0-0] [lt=12] the tablet that inner increases ref cnt is(ret=0, hold_ref_cnt_=false, ls_id={id:1001}, tablet_id={id:216328}, table_store_addr_.addr_={[ver=1,mode=0,seq=1456261][2nd=461][3rd=0][offset=651264,size=357,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, storage_schema_addr_.addr_={[ver=1,mode=0,seq=1456261][2nd=461][3rd=0][offset=651621,size=739,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, tablet_addr_={[ver=1,mode=0,seq=1456260][2nd=459][3rd=0][offset=348160,size=744,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, this=0x14751ea31050, macro_iter={macro_info:{entry_block:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, meta_block_info_arr:{cnt:0, arr:null, capacity:0}, data_block_info_arr:{cnt:0, arr:null, capacity:0}, shared_meta_block_info_arr:{cnt:1, arr:0x1475981a8300, capacity:1}, shared_data_block_info_arr:{cnt:2, arr:0x1475981a8330, capacity:2}, is_inited:true}, cur_type:4, target_type:6, is_linked:false}, lbt()=“0xa916688 0x1d73c310 0xad3588c 0xac9aaab 0x1d6d452f 0x1e2c0ef0 0x1e29a56d 0x1e299634 0xa7b3fa8 0xa7a300d 0x260bc515 0x260ba5f2 0x147799ab6a83 0x147799b48890”)
[2026-08-09 11:40:45.863709] INFO [STORAGE] inc_ref_with_macro_iter (ob_tablet.cpp:3400) [8289][T1002_TimerWK6_WriteCkpt][T1002][YB42AC175339-00065894F649FD85-0-0] [lt=51] the tablet that inner increases ref cnt is(ret=0, hold_ref_cnt_=true, ls_id={id:1001}, tablet_id={id:216328}, table_store_addr_.addr_={[ver=1,mode=0,seq=1456261][2nd=461][3rd=0][offset=651264,size=357,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, storage_schema_addr_.addr_={[ver=1,mode=0,seq=1456261][2nd=461][3rd=0][offset=651621,size=739,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}, tablet_addr_={[ver=1,mode=0,seq=1456260][2nd=459][3rd=0][offset=348160,size=744,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, this=0x14751ea31050, macro_iter={macro_info:{entry_block:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, meta_block_info_arr:{cnt:0, arr:null, capacity:0}, data_block_info_arr:{cnt:0, arr:null, capacity:0}, shared_meta_block_info_arr:{cnt:1, arr:0x14751ea31a6c, capacity:1}, shared_data_block_info_arr:{cnt:2, arr:0x14751ea31a8c, capacity:2}, is_inited:true}, cur_type:4, target_type:6, is_linked:false}, lbt()=“0xa916688 0x1d73c310 0xad3588c 0x1d6f852c 0x1d6dbb44 0x1e2c0f98 0x1e29a56d 0x1e299634 0xa7b3fa8 0xa7a300d 0x260bc515 0x260ba5f2 0x147799ab6a83 0x147799b48890”)
[2026-08-09 11:40:45.863785] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9698][T1002_L0_G0][T1002][YB42AC175339-00065894F9EA02E4-0-0] [lt=0] mvcc_write conflict(key={“INT”:11}, tx_id={txid:20000912}, conflict_tx_id={txid:20000686}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1786177034366336 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246816942065 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20000912} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=20290, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(20291,1,0,0,20291,20290,20290,0)|]}})
[2026-08-09 11:40:45.863863] INFO [STORAGE] sn_async_write (ob_storage_object_handle.cpp:301) [8289][T1002_TimerWK6_WriteCkpt][T1002][YB42AC175339-00065894F649FD85-0-0] [lt=24] Async write macro block(macro_id_={[ver=1,mode=0,seq=1456261][2nd=461]})
[2026-08-09 11:40:45.863902] INFO [STORAGE] reset (ob_tablet.cpp:243) [8084][T1002_TimerWK2_SSTableDefragme][T1002][Y0-0000000000000000-0-0] [lt=11] reset tablet(this=0x1476d5c46060, ls_id={id:-1}, tablet_id={id:0}, lbt()=“0xa916688 0x1d6f3aa3 0x1d6f31c5 0xa8301fe 0xa8d6435 0x1df8cb76 0xa7b3fa8 0xa7a300d 0x260bc515 0x260ba5f2 0x147799ab6a83 0x147799b48890”)
[2026-08-09 11:40:45.863947] WDIAG [STORAGE] get_meta_obj_with_external_memory (ob_tablet_pointer_map.cpp:658) [7454][TimerWK1_BlkMgr][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] add macro ref cnt in write lock, may poor performance(key={ls_id:{id:1003}, tablet_id:{id:1152921504606852313}}, disk_addr={[ver=1,mode=0,seq=1389828][2nd=140480][3rd=0][offset=827392,size=702,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]})
[2026-08-09 11:40:45.864178] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9775][T1002_L0_G0][T1002][YB42AC175339-00065894F9FA03C4-0-0] [lt=0] mvcc_write conflict(key={“INT”:4}, tx_id={txid:20000905}, conflict_tx_id={txid:20000866}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1786177034366336 max_table_version=1786177034366336 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246832022405 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20000905} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=9560, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(9561,1,0,0,9561,9560,9560,0)|]}})
[2026-08-09 11:40:45.864160] INFO [STORAGE] deserialize (ob_tablet.cpp:3082) [8147][T1002_TransferS][T1002][YB42AC175339-00065894F919FD4A-0-0] [lt=1] succeed to load deserialize tablet(ret=0, this={this:0x14751ea63050, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:0, version:4, length:519, tablet_addr:{[ver=1,mode=0,seq=1351373][2nd=91537][3rd=0][offset=1826816,size=708,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, allocator:null, tablet_meta:{version:1, ls_id:{id:1002}, tablet_id:{id:1152921504606911709}, data_tablet_id:{id:232591}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:1786189715232029001, v:0}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1786213022146014000, multi_version_start:1786212035185605000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1786212002695335000, cur_report_version:1786212002695335000, data_checksum:0, row_count:0, found_cg_checksum_error:false}, table_store_flag:{with_major_sstable:1, is_user_data_table:0}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1786242984803836000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, extra_medium_info:{info:4353, compat:1, last_compaction_type:1, wait_check_flag:1, last_medium_scn:1786212002695335000}, last_persisted_committed_tablet_status:{tablet_status:{val:1, str:“NORMAL”}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1786189715246790001, v:0}, create_commit_version:1786189715246790001, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, start_transfer_commit_version:-1, start_split_commit_version:-1}, create_schema_version:1786189715080864, space_usage:{all_sstable_data_occupy_size:493, all_sstable_data_required_size:0, all_sstable_meta_size:0, ss_public_sstable_occupy_size:0, tablet_clustered_meta_size:8192, tablet_clustered_sstable_data_size:4096, backup_bytes:0}, micro_index_clustered:false, ddl_replay_status:14, split_info:{split_info:0, split_src_tablet_id:{id:0}}, has_truncate_info:false}, table_store_addr:{ptr:0x14751ea636c0, addr:{[ver=1,mode=0,seq=1351619][2nd=91840][3rd=0][offset=1171456,size=354,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}}, storage_schema_addr:{ptr:null, addr:{[ver=1,mode=0,seq=1351619][2nd=91840][3rd=0][offset=1171810,size=265,type=2,seq=0][trans_seq=0, sec_part=0][6th=0]}}, macro_info_addr:{ptr:0x14751ea63ce0, addr:{[ver=1,mode=0,seq=1351373][2nd=91537][3rd=0][offset=1827356,size=168,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}}, mds_data:null, ddl_kvs:null, ddl_kv_count:0, is_external_tablet:false, table_store_cache:{last_major_snapshot_version:1786212002695335000, major_table_cnt:1, minor_table_cnt:0, recycle_version:1786212002695335000, last_major_column_count:8, last_major_macro_block_cnt:0, is_row_store:true, is_tablet_referenced_by_collect_mv:false, last_major_compressor_type:6, last_major_latest_row_store_type:1, last_major_store_type:1}, rowkey_read_info:0x14751ea634b0, is_cs_replica_compat:false, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}, header={version:3, length:519, checksum:2016951083, desc_array:[{type:0, length:168}]})
[2026-08-09 11:40:45.864325] INFO [STORAGE.TRANS] rollback_to_implicit_savepoint (ob_tx_api.cpp:1116) [9769][T1002_L0_G0][T1002][YB42AC175339-00065894F9FA03C3-0-0] [lt=0] add tx part with extra_touched_ls(extra_touched_ls=[{id:1002}], tx.tx_id={txid:20000904})
[2026-08-09 11:40:45.864541] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9698][T1002_L0_G0][T1002][YB42AC175339-00065894F9EA02E4-0-0] [lt=0] mvcc_write conflict(key={“INT”:11}, tx_id={txid:20000912}, conflict_tx_id={txid:20000686}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1786177034366336 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246816942065 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20000912} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=20295, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(20296,1,0,0,20296,20295,20295,0)|]}})
[2026-08-09 11:40:45.864513] INFO [STORAGE] assign_pointer_handle (ob_tablet.cpp:5128) [8084][T1002_TimerWK2_SSTableDefragme][T1002][Y0-0000000000000000-0-0] [lt=10] is_external_tablet, should not hold tablet_pointer(ret=0, ptr_hdl={ptr:{ptr:{phy_addr:{[ver=1,mode=0,seq=1407102][2nd=10932][3rd=0][offset=794624,size=706,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]}, obj:{pool:0x1477320f5930, allocator:null, ptr:null, t3m:0x1477320c4030}, ls_handle:{ls_map_:0x1477321ee050, ls_:0x14770813a150, mod_:13}, ddl_kv_mgr_handle:{obj:null, obj_pool:null, allocator:null, t3m:null}, attr:{valid:false, is_empty_shell:false, has_transfer_table:false, has_next_tablet:false, has_nested_table:false, ha_status:0, all_sstable_data_occupy_size:0, all_sstable_data_required_size:0, tablet_meta_size:0, ss_public_sstable_occupy_size:0, backup_bytes:0}, protected_memtable_mgr_handle:{memtable_mgr_handle_:{memtable_mgr:null, pool:null}}, ddl_info:{ddl_schema_version:0, ddl_schema_refreshed_ts:-1, schema_version_change_scn:{val:0, v:0}}, initial_state:false, old_version_chain:null, flying:false}, ref_cnt:3}, map:0x1477320c42f0}, this=0x1476d5c46060, lbt()=“0xa916688 0x1d752ea6 0xa91cce6 0x1c9ae253 0xa830230 0xa8d6435 0x1df8cb76 0xa7b3fa8 0xa7a300d 0x260bc515 0x260ba5f2 0x147799ab6a83 0x147799b48890”)
[2026-08-09 11:40:45.864633] WDIAG [STORAGE] get_meta_obj_with_external_memory (ob_tablet_pointer_map.cpp:658) [8084][T1002_TimerWK2_SSTableDefragme][T1002][Y0-0000000000000000-0-0] [lt=70][errcode=0] add macro ref cnt in write lock, may poor performance(key={ls_id:{id:1003}, tablet_id:{id:1152921504606852414}}, disk_addr={[ver=1,mode=0,seq=1407102][2nd=10932][3rd=0][offset=794624,size=706,type=4,seq=0][trans_seq=0, sec_part=0][6th=0]})
[2026-08-09 11:40:45.864887] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9775][T1002_L0_G0][T1002][YB42AC175339-00065894F9FA03C4-0-0] [lt=0] mvcc_write conflict(key={“INT”:4}, tx_id={txid:20000905}, conflict_tx_id={txid:20000866}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1786177034366336 max_table_version=1786177034366336 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246832022405 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20000905} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=9565, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(9566,1,0,0,9566,9565,9565,0)|]}})
[2026-08-09 11:40:45.865146] INFO [DETECT] maintain_deadlock_info_when_end_stmt (ob_trans_deadlock_adapter.cpp:649) [9698][T1002_L0_G0][T1002][YB42AC175339-00065894F9EA02E4-0-0] [lt=1] maintain deadlock info(step=10, ret=0, ret=“OB_SUCCESS”, exec_ctx.get_errcode()=-6005, exec_ctx.get_errcode()=“OB_TRY_LOCK_ROW_CONFLICT”, session={this:0x1475fe2e7518, id:3221517930, client_sid:4294967295, deser:false, tenant:“mytenant”, tenant_id:1002, effective_tenant:“mytenant”, effective_tenant_id:1002, database:“qianyemarket”, user:“qianyemarket@%”, consistency_level:3, session_state:2, autocommit:true, tx:0x1476b4410be0}, desc={this:0x1476b4410be0, tx_id:{txid:20000912}, state:2, addr:“172.23.83.57:2882”, tenant_id:1002, session_id:3221517930, assoc_session_id:3221517930, client_sid:3221517930, xid:NULL, xa_mode:"", xa_start_addr:“0.0.0.0:0”, access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:{branch:0, seq:1}, op_sn:60901, alloc_ts:1786246653200444, active_ts:1786246653200444, commit_ts:-1, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1786333053200444, coord_id:{id:-1}, parts:[{id:{id:1001}, addr:“172.23.83.57:2882”, epoch:2539889527141990, first_scn:{branch:0, seq:163741728}, last_scn:{branch:0, seq:192664105}, last_touch_ts:1786246816942196, flag_.flag_bit_:{is_dup_ls:0, is_clean:false}}], exec_info_reap_ts:1786246816942195, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1720306770, cluster_version:17180067078, seq_base:1786246653200454, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], last_rc_snapshot_version:{val:1786246845864499002, v:0}, ref:2}, is_rollback=true, conflict_txs=[{tx_id:{txid:20000686}, scheduler_addr:“172.23.83.57:2882”}])
[2026-08-09 11:40:45.865324] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9698][T1002_L0_G0][T1002][YB42AC175339-00065894F9EA02E4-0-0] [lt=0] mvcc_write conflict(key={“INT”:11}, tx_id={txid:20000912}, conflict_tx_id={txid:20000686}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=0 max_table_version=1786177034366336 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246816942065 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20000912} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=20300, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(20301,1,0,0,20301,20300,20300,0)|]}})
[2026-08-09 11:40:45.865406] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9411][T1002_L0_G0][T1002][YB42AC175339-00065894F9EA02C0-0-0] [lt=0] mvcc_write conflict(key={“INT”:4}, tx_id={txid:20000844}, conflict_tx_id={txid:20000866}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1786177034366336 max_table_version=1786177034366336 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246763675999 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20000844} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=79035, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(79036,1,0,0,79036,79035,79035,0)|]}})
[2026-08-09 11:40:45.865428] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9735][T1002_L0_G0][T1002][YB42AC175339-00065894FA2A033A-0-0] [lt=0] mvcc_write conflict(key={“INT”:5}, tx_id={txid:20000933}, conflict_tx_id={txid:20000846}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1786177128858904 max_table_version=1786177128858904 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246824981935 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20000933} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=13180, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(13181,1,0,0,13181,13180,13180,0)|]}})
[2026-08-09 11:40:45.865572] INFO [STORAGE.TRANS] on_wlock_retry (ob_memtable_context.cpp:306) [9663][T1002_L0_G0][T1002][YB42AC175339-00065894F81A039F-0-0] [lt=1] mvcc_write conflict(key={“INT”:5}, tx_id={txid:20000878}, conflict_tx_id={txid:20000846}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1786177128858904 max_table_version=1786177128858904 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=1786246808899524 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:20000878} ls_id=1001 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=27360, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(27361,1,0,0,27361,27360,27360,0)|]}})

7 个赞

不是死锁。
是一堆事务在抢同一行数据的写锁,抢不到就一直重试,最后超时回滚,业务就被堵死了。数据库本身没宕机,是被"热点行"卡住了。
三步把:1、先确认是不是真死锁,跑一句 SELECT * FROM oceanbase.DBA_OB_DEADLOCK_EVENT_HISTORY; 啥也没出来,那基本就是普通的锁打架,不是死锁。2、恢复业务,去 GV$OB_LOCKS 里找 block=1 的持锁事务,顺藤摸瓜在 GV$OB_PROCESSLIST 里定位到会话,一个 ALTER SYSTEM KILL SESSION 把它干掉,业务立马就活。3、彻底解决,看日志里那几个 INT:4/5/11/16,典型的热点行(像库存、余额这种),办法是把一行拆多行、事务别拖太长、更新顺序统一。光把锁超时调大只是拖时间,治不了本。

7 个赞

SELECT * FROM oceanbase.DBA_OB_DEADLOCK_EVENT_HISTORY确实没有记录。你的猜测没错
遗憾的是这会儿CPU已经满了。磁盘的读IO也被打满了, 根本执行不了任何SQL。。。
observer.log.20260809143252991.7z (10.6 MB)
observer.log.20260809143455136.7z (6.0 MB)
observer.log.20260809143904726.7z (5.5 MB)

下午2点半也有一次。这是集群的三个节点的日志,有兴趣的话可以帮忙看看,大大感谢

4 个赞

好多

2 个赞

感谢分享

2 个赞

支持支持

2 个赞

你这个问题的深度可以。

3 个赞

写得真详细,学习了!

2 个赞

OceanBase 社区已接收您的帖子,正在跟进中。

2 个赞

感谢分享,很有参考价值!

1 个赞

主要问题是热点主键 INT‑4、INT‑5、INT‑11 发生大量并发写锁争抢、行锁冲突,大量事务不断触发 on_wlock_retry 写锁重试,部分事务最终抛出 OB_TRY_LOCK_ROW_CONFLICT(‑6005) 锁冲突失败、事务回滚;附带后台合并、分片重置、宏块内存锁带来的性能隐患

从ash报告+日志 来看的话
归档.zip (50.6 KB)

[10:26 上午 ASH 前兆]
ErpSaleGoods FULL/RANGE SCAN + count(distinct)/CTE + PX
→ 60% db file data read + DTL 等待

[14:20~14:32 下午 ASH + 日志]
同一类 SQL 延续;TFSwap 狂写临时文件(ASH: 18GB+ @147MB/s)
→ tmp_block_cache~15GB + WBP 打满 → -9124
→ DTL/PX 变慢;clog palf write 变慢(-4389)

[14:34~14:40 雪崩 — 业务不可用]
worker 被慢 SQL/IO/锁占满
→ wait in request queue 占 80%~94%,AAS 从 142 → 742
→ 新请求几乎无法得到执行 → “集群不可用

综合就是上午大扫表 → 下午临时文件打爆内存/IO → 14:34 起请求队列堆死 → 业务感觉集群不可用 整体是这样的原因导致的
需要根据ASH报告 优化一些查询语句和行锁冲突的语句