[2025-03-19 15:02:40.005991] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [784638][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-394049580236, global_cache_size=21694087168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1003, 1004]) [2025-03-19 15:02:40.009398] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [801854][T1004_BalanceEx][T1004][YB420A686AAD-00062F0A7D55A0CF-0-0] [lt=14] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-03-19 15:02:40.009438] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [801854][T1004_BalanceEx][T1004][YB420A686AAD-00062F0A7D55A0CF-0-0] [lt=37] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-03-19 15:02:40.011768] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:40.011936] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [785068][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=26] replay status is not follower(min_unreplayed_lsn={lsn:25887514}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1740557427596568, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:25887514}, next_to_submit_scn_:{val:1740557386475400001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1740557196095704001, v:0}, iterator_:{iterator_impl:{buf_:0x7effbba05000, next_round_pread_size:3678208, curr_read_pos:582, curr_read_buf_start_pos:0, curr_read_buf_end_pos:582, log_storage_:{start_lsn:{lsn:25886932}, end_lsn:{lsn:25887514}, read_buf:{buf_len_:3747840, buf_:0x7effbba05000}, block_size:67104768, log_storage_:0x7effbcbc9a70, log_storage_:{log_tail:{lsn:10985015063}, readable_log_tail:{lsn:10985015063}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:10938077184}, min_scn:{val:1742359108179818001, v:0}, curr_block_id:163, palf_id:1}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1/1/log", dir_fd:300, min_block_id:0, max_block_id:164, curr_writable_block_id:163}, logical_block_size_:67104768, curr_block_writable_size_:20166889, block_header_serialize_buf_:0x7effbcbca738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1740557386475400000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:494, scn_:{val:1740557386475400000, v:0}, data_checksum:4137500652, flag:59868}}, init_mode_version:0, accumulate_checksum:3745034526, curr_entry_is_padding:0, padding_entry_size:526, padding_entry_scn:{val:1740557386475400000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:9, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:2, miss_cnt:9, cache_read_size:2032, hit_ratio:"1.818181818181818232e-01"}, read_io_cnt:11, read_io_size:26099712, read_disk_cost_ts:41916, start_lsn:{lsn:0}}}}}}) [2025-03-19 15:02:40.012058] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [785068][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=120] get_replay_process success(id={id:1}, submitted_log_size=25887514, unsubmitted_log_size=0, replayed_log_size=25887514, unreplayed_log_size=0) [2025-03-19 15:02:40.012114] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [785068][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=28] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=24, replayed_log_size(MB)=24, last_submitted_log_size(MB)=24, last_replayed_log_size(MB)=24, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-03-19 15:02:40.028794] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [801946][T1004_LSArchive][T1004][YB420A686AAD-00062F086E4100C5-0-0] [lt=13] gc stale ls task succ [2025-03-19 15:02:40.028857] INFO [ARCHIVE] add_ls_task_ (ob_ls_mgr.cpp:456) [801946][T1004_LSArchive][T1004][YB420A686AAD-00062F086E4100C5-0-0] [lt=40] ls not need do archive on this server(ls_id={id:1002}) [2025-03-19 15:02:40.028899] INFO [ARCHIVE] add_ls_task_ (ob_ls_mgr.cpp:456) [801946][T1004_LSArchive][T1004][YB420A686AAD-00062F086E4100C5-0-0] [lt=39] ls not need do archive on this server(ls_id={id:1003}) [2025-03-19 15:02:40.050440] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [785215][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=26] ls blacklist refresh finish(cost_time=13858) [2025-03-19 15:02:40.067780] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [784884][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=36] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.068283] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784870][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.068355] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784875][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.068596] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784877][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.068725] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784873][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.068728] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784872][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.068729] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784869][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.068732] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784876][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.068789] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784874][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.068850] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784871][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.076730] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784881][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.076730] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784880][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.076729] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784878][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.076784] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784879][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.076786] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784882][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.077220] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784886][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.086564] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=12] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.104.106.173:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-03-19 15:02:40.086595] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.104.106.173:2882", has_set_deleted:false}, append size=1220) [2025-03-19 15:02:40.111829] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:40.118929] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAD-00062F0B1BE93B34-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9, this=0x7efcf40b96a0, extra_info="palf_id:1002") [2025-03-19 15:02:40.118961] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAD-00062F0B1BE93B34-0-0] [lt=31] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=579, this=0x7efcf40b98d0, extra_info="palf_id:1002") [2025-03-19 15:02:40.119733] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAD-00062F0B1BE93B35-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x7efdb10d36a0, extra_info="palf_id:1003") [2025-03-19 15:02:40.119772] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAD-00062F0B1BE93B35-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=568, this=0x7efdb10d38d0, extra_info="palf_id:1003") [2025-03-19 15:02:40.124077] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [785309][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=31] current monitor number(seq_=-1) [2025-03-19 15:02:40.153565] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [785326][T1_L0_G25][T1][YB420A686AAD-00062F071C9ECF96-0-0] [lt=35] [RS_MGR] force set rs list(master_rs="10.104.106.173:2882") [2025-03-19 15:02:40.153604] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [785326][T1_L0_G25][T1][YB420A686AAD-00062F071C9ECF96-0-0] [lt=34] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"10.104.106.173:2882", server_id:1, rs_addr:"10.104.106.173:2882", rs_server_status:1, epoch_id:2}, hb_response={zone:"zone1", server:"10.104.106.173:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1740557429500476, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=41) [2025-03-19 15:02:40.158451] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [802166][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=51] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.104.106.173:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=12984, avg_group_log_size=811) [2025-03-19 15:02:40.158489] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [802166][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.104.106.173:2882", has_set_deleted:false}, append size=12984) [2025-03-19 15:02:40.158686] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [802139][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=37] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.104.106.173:2882", committed size=12395) [2025-03-19 15:02:40.159016] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [2924099][T1003_L0_G1][T1003][YB420A686AAE-00062F08B2136A2E-0-0] [lt=18] [PALF STAT LOG TASK TIME](palf_id=1, self="10.104.106.173:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=4, avg_log_gen_to_submit_time=57, avg_log_submit_to_flush_time=260, avg_log_submit_to_slide_time=567) [2025-03-19 15:02:40.160101] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [802132][T1003_TimerWK3][T1003][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1003, evict_high_mem_level=103079214, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=95668224) [2025-03-19 15:02:40.160167] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [802131][T1003_TimerWK2][T1003][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=1659, get_cur_idx()=1651, get_pop_idx()=1651) [2025-03-19 15:02:40.177077] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=81] sql audit evict task end(request_manager_->get_tenant_id()=1004, evict_high_mem_level=927712935, evict_high_size_level=884736, evict_batch_count=0, elapse_time=1, size_used=458752, mem_used=900529152) [2025-03-19 15:02:40.177132] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [801889][T1004_TimerWK0][T1004][Y0-0000000000000000-0-0] [lt=57] Construct Queue Num(construct_num=0, get_push_idx()=974, get_cur_idx()=966, get_pop_idx()=959) [2025-03-19 15:02:40.183384] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [784638][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=5) [2025-03-19 15:02:40.183434] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [802189][T1003_L5_G0][T1003][YB420A686AAF-00062F0830C795D4-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1742367760182510000, start_id=1742367760182510000, end_id=1742367760182510001) [2025-03-19 15:02:40.186547] INFO [LIB] stat (utility.h:1210) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=47, this=0x7efe9d9a7470, extra_info="") [2025-03-19 15:02:40.186587] INFO [LIB] stat (utility.h:1210) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x7efdd2dcc050, extra_info="id:1") [2025-03-19 15:02:40.186761] INFO try_report (ob_tc_stat.cpp:35) [784913][][T0][Y0-0000000000000000-0-0] [lt=32] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:6,x,0 t1m0gd:1,x,0 t1m1:6,x,0 t1m1gd:1,x,0 t1m2:6,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1m0g10000:100,x,0 t1m1g10000:100,x,0 t1m2g10000:100,x,0 t1002m0:6,x,0 t1002m0gd:1,x,0 t1002m1:6,x,0 t1002m1gd:1,x,0 t1002m2:6,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:6,x,0 t1001m0gd:1,x,0 t1001m1:6,x,0 t1001m1gd:1,x,0 t1001m2:6,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x t1004m0:12,x,0 t1004m0gd:1,x,0 t1004m1:12,x,0 t1004m1gd:1,x,0 t1004m2:12,x,x t1004m2gd:1,x,0 t1004m0g0:100,x,0 t1004m1g0:100,x,0 ops:x obw:x ops:x obw:x t1004m2g0:100,x,x t1003m0:12,x,0 t1003m0gd:1,x,0 t1003m1:12,x,0 t1003m1gd:1,x,0 t1003m2:12,x,x t1003m2gd:1,x,0 t1003m0g0:100,x,0 ips:x ibw:x t1003m1g0:100,x,0 ops:x obw:x t1003m2g0:100,x,x|stat: root:316K/74:14138 disk:316K/74:14138 t1m2:52K/12:13233 t1m2g0:52K/12:13233 t1004m2:180K/44:14058 t1004m2g0:180K/44:14058 t1003m2:84K/18:14934 t1003m2g0:84K/18:14934 |q0: 0 0 0 |q1: 0 0 0 [2025-03-19 15:02:40.186785] INFO [LIB] stat (utility.h:1210) [801900][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=223, this=0x7efe9d9b1c30, extra_info="palf_id:1") [2025-03-19 15:02:40.186826] INFO [LIB] stat (utility.h:1227) [801900][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=33] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x7efd20dfe530, extra_info="") [2025-03-19 15:02:40.186846] INFO [LIB] stat (utility.h:1210) [801899][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9, this=0x7efe9d9b1e60, extra_info="palf_id:1") [2025-03-19 15:02:40.187172] INFO [LIB] stat (utility.h:1210) [2924074][T1004_L0_G1][T1004][YB420A686AAE-00062F08B5415EB6-0-0] [lt=29] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=10, this=0x7efe9d9a76a0, extra_info="palf_id:1") [2025-03-19 15:02:40.187206] INFO [LIB] stat (utility.h:1210) [2924074][T1004_L0_G1][T1004][YB420A686AAE-00062F08B5415EB6-0-0] [lt=33] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=658, this=0x7efe9d9a78d0, extra_info="palf_id:1") [2025-03-19 15:02:40.187292] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [802111][T1003_Occam][T1003][YB420A686AAD-00062F089AD100C6-0-0] [lt=38] ====== tenant freeze timer task ====== [2025-03-19 15:02:40.189265] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [802140][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=16] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.104.106.173:2882", has_set_deleted:false}, accum_size=14074) [2025-03-19 15:02:40.192011] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [785109][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.104.106.173:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=5499, avg_group_log_size=458) [2025-03-19 15:02:40.192045] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [785109][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.104.106.173:2882", has_set_deleted:false}, append size=5499) [2025-03-19 15:02:40.203243] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [802167][T1003_LSArchive][T1003][YB420A686AAD-00062F086ED100C5-0-0] [lt=30] gc stale ls task succ [2025-03-19 15:02:40.206498] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [784636][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-394049580236, global_cache_size=21694087168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1003, 1004]) [2025-03-19 15:02:40.211416] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [801892][T1004_TimerWK3][T1004][Y0-0000000000000000-0-0] [lt=33] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-03-19 15:02:40.211472] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [801892][T1004_TimerWK3][T1004][Y0-0000000000000000-0-0] [lt=31] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-03-19 15:02:40.211502] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [801892][T1004_TimerWK3][T1004][Y0-0000000000000000-0-0] [lt=26] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-03-19 15:02:40.211517] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [801892][T1004_TimerWK3][T1004][Y0-0000000000000000-0-0] [lt=14] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-03-19 15:02:40.211529] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [801892][T1004_TimerWK3][T1004][Y0-0000000000000000-0-0] [lt=12] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-03-19 15:02:40.211542] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [801892][T1004_TimerWK3][T1004][Y0-0000000000000000-0-0] [lt=11] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=12671) [2025-03-19 15:02:40.211888] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:40.212571] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [785070][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=826623659, evict_high_size_level=825753, evict_batch_count=0, elapse_time=0, size_used=393216, mem_used=723750912) [2025-03-19 15:02:40.212631] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [785069][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=83] Construct Queue Num(construct_num=0, get_push_idx()=2267, get_cur_idx()=2259, get_pop_idx()=2253) [2025-03-19 15:02:40.219206] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [2924119][T1004_L0_G1][T1004][YB420A686AAF-00062F088628416B-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1003, self="10.104.106.173:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-03-19 15:02:40.219793] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [2924119][T1004_L0_G1][T1004][YB420A686AAD-00062F0B1BE93B37-0-0] [lt=39] [PALF STAT LOG TASK TIME](palf_id=1003, self="10.104.106.173:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=8, avg_log_submit_to_flush_time=239, avg_log_submit_to_slide_time=560) [2025-03-19 15:02:40.219803] INFO [PALF] read (log_cache.cpp:103) [801978][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=33] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-03-19 15:02:40.219843] INFO [PALF] next (palf_iterator.h:183) [801978][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=36] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:12956800, miss_cnt:0, cache_read_size:1579121971, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-03-19 15:02:40.221170] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11] [PALF_DUMP](palf_id=1001, self="10.104.106.173:2882", [SlidingWindow]={palf_id:1001, self:"10.104.106.173:2882", lsn_allocator:{max_log_id:12861475, max_lsn:1570901074, max_scn:{val:1742367759995916001, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:1570901074}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:1528958034}}, last_submit_lsn:{lsn:1570900952}, last_submit_end_lsn:{lsn:1570901074}, last_submit_log_id:12861475, last_submit_log_pid:1, max_flushed_lsn:{lsn:1570900952}, max_flushed_end_lsn:{lsn:1570901074}, max_flushed_log_pid:1, committed_end_lsn:{lsn:1570901074}, last_slide_log_id:12861475, last_slide_scn:{val:1742367759995916001, v:0}, last_slide_lsn:{lsn:1570900952}, last_slide_end_lsn:{lsn:1570901074}, last_slide_log_pid:1, last_slide_log_accum_checksum:1734964512, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7efd7e8fe530}, [StateMgr]={this:0x7efd7e902830, self:"10.104.106.173:2882", palf_id:1001, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"10.104.106.173:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:12861476, is_changing_config_with_arb:false, reconfirm_start_time_us:1741079490510886, palf_role_change_cb:0x7efd7e9092f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"10.104.106.173:2882", alive_paxos_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:1, config_seq:9}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:1, lock_owner:8, lock_time:1741317367277019}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:10}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:8, lock_time:1741317367648033}}, prev_log_proposal_id:1, prev_lsn:{lsn:293620195}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:8, lock_type:0, added_list:0, removed_list:0, type:"UNLOCK_CONFIG_CHANGE"}, state:0, checking_barrier:{prev_log_proposal_id:1, prev_lsn:{lsn:293620195}, prev_end_lsn:{lsn:293620365}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:1, prev_lsn:{lsn:293620195}, prev_end_lsn:{lsn:293620365}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:10}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:10}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7efd7e8ffc70}, [ModeMgr]={palf_id:1001, self:"10.104.106.173:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476753765000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476753765000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476753765000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1741079490711038, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1741079476753765001, v:0}, base_lsn_for_block_gc:{lsn:1476304896}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:1, config_seq:9}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:1, lock_owner:8, lock_time:1741317367277019}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:10}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:8, lock_time:1741317367648033}}, prev_log_proposal_id:1, prev_lsn:{lsn:293620195}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:1476304896}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1741079476753765000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476753765000, v:0}}}, log_meta_storage:{log_tail:{lsn:135168}, readable_log_tail:{lsn:135168}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1004/1001/meta", dir_fd:388, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66969600, block_header_serialize_buf_:0x7efd7e904a38, flashback_version:0}, log_storage:{log_tail:{lsn:1570901074}, readable_log_tail:{lsn:1570901074}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1543409664}, min_scn:{val:1742345127605577005, v:0}, curr_block_id:23, palf_id:1001}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1004/1001/log", dir_fd:389, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:39613358, block_header_serialize_buf_:0x7efd7e906738, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1741079490510756, this:0x7efd7e902f70}, [Reconfirm]={palf_id:1001, self:"10.104.106.173:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"10.104.106.175:2882";"10.104.106.174:2882";, curr_paxos_follower_list:2{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"10.104.106.175:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1741079490511303, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7efd7e9029b0}) [2025-03-19 15:02:40.221489] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=315] [PALF_DUMP](palf_id=1001, self="10.104.106.173:2882", ack_info_list=[{member:{server:"10.104.106.173:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760187074, last_flushed_end_lsn:{lsn:1570901074}}, {member:{server:"10.104.106.174:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760187547, last_flushed_end_lsn:{lsn:1570901074}}, {member:{server:"10.104.106.175:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760187238, last_flushed_end_lsn:{lsn:1570901074}}]) [2025-03-19 15:02:40.221517] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=25] current log_task status(palf_id=1001, self="10.104.106.173:2882", log_id=12861476, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"10.104.106.173:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760187074, last_flushed_end_lsn:{lsn:1570901074}}, {member:{server:"10.104.106.174:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760187547, last_flushed_end_lsn:{lsn:1570901074}}, {member:{server:"10.104.106.175:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760187238, last_flushed_end_lsn:{lsn:1570901074}}]) [2025-03-19 15:02:40.264459] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [2922957][T1004_L0_G18][T1004][YB420A686AAE-00062F085E75A021-0-0] [lt=39] start get_ls_replayed_scn(arg={tenant_id:1004, ls_id:{id:1002}, all_replica:false}) [2025-03-19 15:02:40.264506] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [2922957][T1004_L0_G18][T1004][YB420A686AAE-00062F085E75A021-0-0] [lt=36] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1742367759976931002, v:0}, arg={tenant_id:1004, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1004, ls_id:{id:1002}, cur_readable_scn:{val:1742367759976931002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"10.104.106.173:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-03-19 15:02:40.266879] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [785554][T1_L0_G10000][T1][YB420A686AAD-00062F095B2F3E53-0-0] [lt=76] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:10006, add_timestamp:1742367760266858}) [2025-03-19 15:02:40.266926] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EBB-0-0] [lt=37] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=10006, locations=[{table_id:10006, partition_id:10006, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760266904, sql_renew_time:1742367760266904, is_mark_fail:false}]) [2025-03-19 15:02:40.266974] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EBB-0-0] [lt=47] success to process renew task(task={tenant_id:1, table_id:10006, add_timestamp:1742367760266858}, locations=[{table_id:10006, partition_id:10006, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760266904, sql_renew_time:1742367760266904, is_mark_fail:false}]) [2025-03-19 15:02:40.267143] WDIAG [SHARE] inner_to_show_str (ob_system_variable.cpp:516) [785554][T1_L0_G10000][T1][YB420A686AAD-00062F095B2F3E53-0-0] [lt=37][errcode=0] sys var casted obj is not ObVarcharType(ret=0, value={"NULL":"NULL"}, *res_obj={"NULL":"NULL"}, get_name()=secure_file_priv) [2025-03-19 15:02:40.267367] WDIAG [SHARE] inner_to_show_str (ob_system_variable.cpp:516) [785554][T1_L0_G10000][T1][YB420A686AAD-00062F095B2F3E53-0-0] [lt=29][errcode=0] sys var casted obj is not ObVarcharType(ret=0, value={"NULL":"NULL"}, *res_obj={"NULL":"NULL"}, get_name()=external_user) [2025-03-19 15:02:40.267987] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [802153][T1003_TFSwap][T1003][Y0-0000000000000000-0-0] [lt=39] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-03-19 15:02:40.268028] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [802153][T1003_TFSwap][T1003][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-03-19 15:02:40.268046] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [802153][T1003_TFSwap][T1003][Y0-0000000000000000-0-0] [lt=16] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-03-19 15:02:40.268070] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [802153][T1003_TFSwap][T1003][Y0-0000000000000000-0-0] [lt=21] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-03-19 15:02:40.268092] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [802153][T1003_TFSwap][T1003][Y0-0000000000000000-0-0] [lt=22] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:2248748583528, flush_io_finished_ret_:0, flush_io_finished_round_:1288098, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-03-19 15:02:40.271379] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [785554][T1_L0_G10000][T1][YB420A686AAD-00062F095B2F3E56-0-0] [lt=22] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12095, add_timestamp:1742367760271368}) [2025-03-19 15:02:40.271404] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EBC-0-0] [lt=26] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12095, locations=[{table_id:12095, partition_id:12095, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760271392, sql_renew_time:1742367760271392, is_mark_fail:false}]) [2025-03-19 15:02:40.271440] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EBC-0-0] [lt=35] success to process renew task(task={tenant_id:1, table_id:12095, add_timestamp:1742367760271368}, locations=[{table_id:12095, partition_id:12095, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760271392, sql_renew_time:1742367760271392, is_mark_fail:false}]) [2025-03-19 15:02:40.279712] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-03-19 15:02:40.279788] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:53113, [delay/us]:prepare:6, schedule:17, submit:19, rt:117, total:159) [2025-03-19 15:02:40.279814] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:6, memory_limit_:21474836480, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:6, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:6}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 10000:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:0}, group_id = 10000:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:0}, group_id = 10000:{mode:2, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:0}]}, hold_mem=35100256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0, 0, 0, 0, 0, 0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=3, iops=3, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-03-19 15:02:40.279896] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=64] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:53113, [delay/us]: prepare:6, schedule:17, submit:19, device:117, total:159) [2025-03-19 15:02:40.279922] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:340, idx_:0, tid_:785072}, {is_inited_:true, queue_.get_total():0, tg_id_:284, idx_:1, tid_:785339}, {is_inited_:true, queue_.get_total():0, tg_id_:347, idx_:2, tid_:96558}, {is_inited_:true, queue_.get_total():0, tg_id_:348, idx_:3, tid_:96562}, {is_inited_:true, queue_.get_total():0, tg_id_:349, idx_:4, tid_:96565}, {is_inited_:true, queue_.get_total():0, tg_id_:350, idx_:5, tid_:96569}], io_callback_status=runner 0: thread_id=785072, queue_count=0, runner 1: thread_id=785339, queue_count=0, runner 2: thread_id=96558, queue_count=0, runner 3: thread_id=96562, queue_count=0, runner 4: thread_id=96565, queue_count=0, runner 5: thread_id=96569, queue_count=0, ) [2025-03-19 15:02:40.279994] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=31] [IO STATUS GROUP SYS](tenant_id=1003, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4818, iops:17, bw:81713, [delay/us]:prepare:5, schedule:19, submit:19, rt:147, total:192) [2025-03-19 15:02:40.280011] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1003, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:4294967296, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:12, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:12}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31768544, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0, 0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-03-19 15:02:40.280063] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS FUNCTION](tenant_id=1003, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4819, avg_iops:17, avg_bw:81714, [delay/us]: prepare:5, schedule:19, submit:19, device:147, total:192) [2025-03-19 15:02:40.280094] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1003, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:471, idx_:0, tid_:802134}, {is_inited_:true, queue_.get_total():0, tg_id_:477, idx_:1, tid_:802198}], io_callback_status=runner 0: thread_id=802134, queue_count=0, runner 1: thread_id=802198, queue_count=0, ) [2025-03-19 15:02:40.280130] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS GROUP SYS](tenant_id=1004, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16342, [delay/us]:prepare:5, schedule:15, submit:1, rt:131, total:152) [2025-03-19 15:02:40.280168] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP SYS](tenant_id=1004, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4198, iops:40, bw:167513, [delay/us]:prepare:7, schedule:18, submit:26, rt:141, total:193) [2025-03-19 15:02:40.280183] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1004, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:38654705664, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:12, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:12}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=38170848, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-03-19 15:02:40.280225] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=37] [IO STATUS FUNCTION](tenant_id=1004, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16343, [delay/us]: prepare:5, schedule:15, submit:1, device:131, total:152) [2025-03-19 15:02:40.280253] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS FUNCTION](tenant_id=1004, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4198, avg_iops:40, avg_bw:167517, [delay/us]: prepare:7, schedule:18, submit:26, device:141, total:193) [2025-03-19 15:02:40.280277] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1004, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:403, idx_:0, tid_:801894}, {is_inited_:true, queue_.get_total():0, tg_id_:411, idx_:1, tid_:802078}, {is_inited_:true, queue_.get_total():0, tg_id_:412, idx_:2, tid_:802079}, {is_inited_:true, queue_.get_total():0, tg_id_:413, idx_:3, tid_:802080}, {is_inited_:true, queue_.get_total():0, tg_id_:414, idx_:4, tid_:802081}, {is_inited_:true, queue_.get_total():0, tg_id_:415, idx_:5, tid_:802082}, {is_inited_:true, queue_.get_total():0, tg_id_:416, idx_:6, tid_:802083}, {is_inited_:true, queue_.get_total():0, tg_id_:417, idx_:7, tid_:802084}, {is_inited_:true, queue_.get_total():0, tg_id_:418, idx_:8, tid_:802085}, {is_inited_:true, queue_.get_total():0, tg_id_:419, idx_:9, tid_:802086}], io_callback_status=runner 0: thread_id=801894, queue_count=0, runner 1: thread_id=802078, queue_count=0, runner 2: thread_id=802079, queue_count=0, runner 3: thread_id=802080, queue_count=0, runner 4: thread_id=802081, queue_count=0, runner 5: thread_id=802082, queue_count=0, runner 6: thread_id=802083, queue_count=0, runner 7: thread_id=802084, queue_count=0, runner 8: thread_id=802085, queue_count=0, runner 9: thread_id=802086, queue_count=0, ) [2025-03-19 15:02:40.280331] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4085, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:237, total:249) [2025-03-19 15:02:40.280370] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-03-19 15:02:40.280370] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [784638][TimerWK2][T1003][Y0-0000000000000000-0-0] [lt=39] trace max work area(auto_calc=true, tenant_max_memory_limit=4294967296, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=214748364, max_wa_memory_size=214748364, tmp_max_wa_memory_size=214748364, pre_mem_target=214748364, remain_memory_size=214748364, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=3748380672, washable_size=-2, max_workarea_memory_size=214748364, max_tenant_memory_size=546586624, wash_ratio=6, tenant_id=1003) [2025-03-19 15:02:40.280421] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4085, [delay/us]: prepare:0, schedule:0, submit:600000000, device:237, total:249) [2025-03-19 15:02:40.280452] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:784903}], io_callback_status=runner 0: thread_id=784903, queue_count=0, ) [2025-03-19 15:02:40.280473] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x7efe5e6b0f50, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-03-19 15:02:40.280489] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7eff3d3b8150, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-03-19 15:02:40.280481] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [784638][TimerWK2][T1003][Y0-0000000000000000-0-0] [lt=50] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=26843545, calc_global_bound_size=1048576000000, mem_target_=214748364, wa_max_memory_size=214748364, profile_cnt_=0, total_memory_size=0, max_wa_size=214748364, avg_bound_size=26843545, max_bound_size=26843545, min_bound_size_=1048576) [2025-03-19 15:02:40.280498] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x7efe5e6b0850, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-03-19 15:02:40.280512] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f00351ee750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-03-19 15:02:40.280510] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [784638][TimerWK2][T1003][Y0-0000000000000000-0-0] [lt=26] timer to calc global bound size(ret=0, global_bound_size_=26843545, manual_calc_cnt_=11, drift_size_=0, pre_drift_size=0, wa_max_memory_size=214748364, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1003, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=26843545, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=214748364, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-03-19 15:02:40.280525] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7eff66f70150, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-03-19 15:02:40.280533] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x7efe5e6b0150, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-03-19 15:02:40.280552] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f00351ef550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-03-19 15:02:40.280568] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [784904][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SERVER] net_in=246kB/s, net_out=260kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=1280000kB/s [2025-03-19 15:02:40.280573] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [784638][TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=22] trace max work area(auto_calc=true, tenant_max_memory_limit=38654705664, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=1932735283, max_wa_memory_size=1932735283, tmp_max_wa_memory_size=1932735283, pre_mem_target=1932735283, remain_memory_size=1932735283, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=14090719232, washable_size=-2, max_workarea_memory_size=1932735283, max_tenant_memory_size=24563986432, wash_ratio=6, tenant_id=1004) [2025-03-19 15:02:40.280644] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [784638][TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=24] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=241591910, calc_global_bound_size=1048576000000, mem_target_=1932735283, wa_max_memory_size=1932735283, profile_cnt_=0, total_memory_size=0, max_wa_size=1932735283, avg_bound_size=241591910, max_bound_size=241591910, min_bound_size_=1048576) [2025-03-19 15:02:40.280666] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [784638][TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=21] timer to calc global bound size(ret=0, global_bound_size_=241591910, manual_calc_cnt_=778, drift_size_=0, pre_drift_size=0, wa_max_memory_size=1932735283, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1004, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=241591910, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=1932735283, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-03-19 15:02:40.287245] INFO [PALF] read (log_cache.cpp:103) [801905][T1004_LogShared][T1004][Y0-0000000000000000-0-0] [lt=36] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-03-19 15:02:40.296760] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [2924003][T1_L0_G1][T1][YB420A686AAE-00062F071C07EBFA-0-0] [lt=19] receive register_req from acceptor, ignore(self="10.104.106.173:2882", palf_id=1, child={server:"10.104.106.174:2882", region:"sys_region", register_time_us:1742367760296854, keepalive_ts:-1}) [2025-03-19 15:02:40.296796] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [2924003][T1_L0_G1][T1][YB420A686AAE-00062F071C07EBFA-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.104.106.173:2882", child={server:"10.104.106.174:2882", region:"sys_region", register_time_us:1742367760296854, keepalive_ts:-1}, is_to_leader=true) [2025-03-19 15:02:40.298553] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [802131][T1003_TimerWK2][T1003][Y0-0000000000000000-0-0] [lt=35] ====== traversal_flush timer task ====== [2025-03-19 15:02:40.299019] INFO [MDS] flush (mds_table_mgr.cpp:195) [802131][T1003_TimerWK2][T1003][YB420A686AAD-00062F0A22899A61-0-0] [lt=32] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:60476}, rec_scn:{val:1742320907353704001, v:0}}, {tablet_id:{id:60379}, rec_scn:{val:1742320907353704002, v:0}}, {tablet_id:{id:50380}, rec_scn:{val:1742320907353704003, v:0}}, {tablet_id:{id:233}, rec_scn:{val:1742320907353704004, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1742320907353704005, v:0}}, {tablet_id:{id:50192}, rec_scn:{val:1742320907353704006, v:0}}, {tablet_id:{id:60350}, rec_scn:{val:1742320907353704007, v:0}}, {tablet_id:{id:60507}, rec_scn:{val:1742320907353704008, v:0}}, {tablet_id:{id:60259}, rec_scn:{val:1742320907353704009, v:0}}, {tablet_id:{id:188}, rec_scn:{val:1742320907353704010, v:0}}, {tablet_id:{id:224}, rec_scn:{val:1742320907353704011, v:0}}, {tablet_id:{id:308}, rec_scn:{val:1742320907353704012, v:0}}, {tablet_id:{id:369}, rec_scn:{val:1742320907353704013, v:0}}, {tablet_id:{id:493}, rec_scn:{val:1742320907353704014, v:0}}, {tablet_id:{id:101011}, rec_scn:{val:1742320907353704015, v:0}}, {tablet_id:{id:101109}, rec_scn:{val:1742320907353704016, v:0}}, {tablet_id:{id:284}, rec_scn:{val:1742320907417143001, v:0}}, {tablet_id:{id:50149}, rec_scn:{val:1742320907418201001, v:0}}, {tablet_id:{id:307}, rec_scn:{val:1742320907420319004, v:0}}, {tablet_id:{id:50240}, rec_scn:{val:1742320907420319005, v:0}}, {tablet_id:{id:50355}, rec_scn:{val:1742320907420319006, v:0}}, {tablet_id:{id:60457}, rec_scn:{val:1742320907426683001, v:0}}, {tablet_id:{id:182}, rec_scn:{val:1742320907426683002, v:0}}, {tablet_id:{id:101098}, rec_scn:{val:1742320907429828001, v:0}}, {tablet_id:{id:60387}, rec_scn:{val:1742320907429828002, v:0}}, {tablet_id:{id:419}, rec_scn:{val:1742320907433002001, v:0}}, {tablet_id:{id:60305}, rec_scn:{val:1742320907433002002, v:0}}, {tablet_id:{id:60297}, rec_scn:{val:1742320907436175001, v:0}}, {tablet_id:{id:60342}, rec_scn:{val:1742320907436175002, v:0}}, {tablet_id:{id:378}, rec_scn:{val:1742320907436175003, v:0}}, {tablet_id:{id:101036}, rec_scn:{val:1742320907442528003, v:0}}, {tablet_id:{id:60005}, rec_scn:{val:1742320907442528004, v:0}}, }, max_consequent_callbacked_scn={val:1742367760188841000, v:0}, *this={this:0x7eff09fd3150, is_inited:true, freezing_scn:{val:1742320803560658000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1003, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1742320803708055001, v:0}, clog_base_lsn:{lsn:13353848832}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1742118204786497000, v:0}, all_id_meta:{id_meta:[{limited_id:1742367774497129000, latest_log_ts:{val:1742367757955987001, v:0}}, {limited_id:23000001, latest_log_ts:{val:1742359877227663001, v:0}}, {limited_id:220000001, latest_log_ts:{val:1742321077556576001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x7eff1a7e4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:15831098, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-03-19 15:02:40.299215] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [802131][T1003_TimerWK2][T1003][YB420A686AAD-00062F0A22899A61-0-0] [lt=190] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-03-19 15:02:40.299860] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [784635][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] try inc thread count(*this={name:IO_SYNC_CH, this:0x7f00351ef170, min_thread_cnt:1, max_thread_cnt:224, running_thread_cnt:0, threads_idle_time:2248200497419, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-03-19 15:02:40.299901] WDIAG [LIB] try_wait (thread.cpp:249) [784635][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-03-19 15:02:40.299921] INFO [LIB] do_thread_recycle (threads.cpp:146) [784635][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=19] try_wait return eagain(this=0x7f00351ef170, thread=0x7eff6fcd98b0) [2025-03-19 15:02:40.299937] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [784635][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:IO_SYNC_CH, this:0x7f00351ef170, min_thread_cnt:1, max_thread_cnt:224, running_thread_cnt:0, threads_idle_time:2248200497419, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-03-19 15:02:40.300289] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [784639][TimerWK3][T1004][Y0-0000000000000000-0-0] [lt=28] get_disk_usage(ret=0, capacity(MB):=110592, used(MB):=16705) [2025-03-19 15:02:40.308986] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [785554][T1_L0_G10000][T1][YB420A686AAD-00062F095B2F3E59-0-0] [lt=30] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12245, add_timestamp:1742367760308975}) [2025-03-19 15:02:40.309011] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EBD-0-0] [lt=22] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12245, locations=[{table_id:12245, partition_id:12245, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760309000, sql_renew_time:1742367760309000, is_mark_fail:false}]) [2025-03-19 15:02:40.309035] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [785554][T1_L0_G10000][T1][YB420A686AAD-00062F095B2F3E59-0-0] [lt=33] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12296, add_timestamp:1742367760309031}) [2025-03-19 15:02:40.309058] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [785554][T1_L0_G10000][T1][YB420A686AAD-00062F095B2F3E59-0-0] [lt=17] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12297, add_timestamp:1742367760309057}) [2025-03-19 15:02:40.309048] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EBD-0-0] [lt=37] success to process renew task(task={tenant_id:1, table_id:12245, add_timestamp:1742367760308975}, locations=[{table_id:12245, partition_id:12245, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760309000, sql_renew_time:1742367760309000, is_mark_fail:false}]) [2025-03-19 15:02:40.309101] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [785554][T1_L0_G10000][T1][YB420A686AAD-00062F095B2F3E59-0-0] [lt=17] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12298, add_timestamp:1742367760309083}) [2025-03-19 15:02:40.309100] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EBE-0-0] [lt=32] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12296, locations=[{table_id:12296, partition_id:12296, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760309092, sql_renew_time:1742367760309092, is_mark_fail:false}]) [2025-03-19 15:02:40.309122] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EBE-0-0] [lt=22] success to process renew task(task={tenant_id:1, table_id:12296, add_timestamp:1742367760309031}, locations=[{table_id:12296, partition_id:12296, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760309092, sql_renew_time:1742367760309092, is_mark_fail:false}]) [2025-03-19 15:02:40.309148] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EBF-0-0] [lt=15] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12297, locations=[{table_id:12297, partition_id:12297, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760309141, sql_renew_time:1742367760309141, is_mark_fail:false}]) [2025-03-19 15:02:40.309170] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EBF-0-0] [lt=22] success to process renew task(task={tenant_id:1, table_id:12297, add_timestamp:1742367760309057}, locations=[{table_id:12297, partition_id:12297, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760309141, sql_renew_time:1742367760309141, is_mark_fail:false}]) [2025-03-19 15:02:40.309195] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EC0-0-0] [lt=15] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12298, locations=[{table_id:12298, partition_id:12298, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760309189, sql_renew_time:1742367760309189, is_mark_fail:false}]) [2025-03-19 15:02:40.309217] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [784773][VTblLocAsyncUp0][T0][YB420A686AAD-00062F0A9D594EC0-0-0] [lt=21] success to process renew task(task={tenant_id:1, table_id:12298, add_timestamp:1742367760309083}, locations=[{table_id:12298, partition_id:12298, partition_cnt:0, replica_locations:[{server:"10.104.106.173:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1742367760309189, sql_renew_time:1742367760309189, is_mark_fail:false}]) [2025-03-19 15:02:40.311947] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:40.321233] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=43] [PALF_DUMP](palf_id=1002, self="10.104.106.173:2882", [SlidingWindow]={palf_id:1002, self:"10.104.106.173:2882", lsn_allocator:{max_log_id:13105900, max_lsn:11925308646, max_scn:{val:1742367759976931003, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:11925308646}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:11883365606}}, last_submit_lsn:{lsn:11925308524}, last_submit_end_lsn:{lsn:11925308646}, last_submit_log_id:13105900, last_submit_log_pid:1, max_flushed_lsn:{lsn:11925308524}, max_flushed_end_lsn:{lsn:11925308646}, max_flushed_log_pid:1, committed_end_lsn:{lsn:11925308646}, last_slide_log_id:13105900, last_slide_scn:{val:1742367759976931003, v:0}, last_slide_lsn:{lsn:11925308524}, last_slide_end_lsn:{lsn:11925308646}, last_slide_log_pid:1, last_slide_log_accum_checksum:3907354920, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7efcf40b8530}, [StateMgr]={this:0x7efcf40bc830, self:"10.104.106.173:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"10.104.106.174:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:13105901, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x7efcf40c32f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"10.104.106.173:2882", alive_paxos_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:1, config_seq:5}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:1, lock_owner:7, lock_time:1741317364577218}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:6}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:7, lock_time:1741317365085451}}, prev_log_proposal_id:1, prev_lsn:{lsn:294420028}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:6}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1742367758320060, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1742367758320060, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7efcf40b9c70}, [ModeMgr]={palf_id:1002, self:"10.104.106.173:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476766497000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476766497000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476766497000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1741079476766497001, v:0}, base_lsn_for_block_gc:{lsn:11810439168}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:1, config_seq:5}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:1, lock_owner:7, lock_time:1741317364577218}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:6}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:7, lock_time:1741317365085451}}, prev_log_proposal_id:1, prev_lsn:{lsn:294420028}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:11810439168}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1741079476766497000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476766497000, v:0}}}, log_meta_storage:{log_tail:{lsn:122880}, readable_log_tail:{lsn:122880}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:11944648704}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1004/1002/meta", dir_fd:422, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66981888, block_header_serialize_buf_:0x7efcf40bea38, flashback_version:0}, log_storage:{log_tail:{lsn:11925308646}, readable_log_tail:{lsn:11925308646}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:11877543936}, min_scn:{val:1742328730960686003, v:0}, curr_block_id:177, palf_id:1002}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1004/1002/log", dir_fd:424, min_block_id:0, max_block_id:178, curr_writable_block_id:177}, logical_block_size_:67104768, curr_block_writable_size_:19340058, block_header_serialize_buf_:0x7efcf40c0738, flashback_version:0}, palf_epoch:3, last_purge_throttling_ts:-1, this:0x7efcf40bcf70}, [Reconfirm]={palf_id:1002, self:"10.104.106.173:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7efcf40bc9b0}) [2025-03-19 15:02:40.321509] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=272][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="10.104.106.173:2882", tmp_server={server:"10.104.106.173:2882", timestamp:1, flag:0}) [2025-03-19 15:02:40.321528] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] [PALF_DUMP](palf_id=1002, self="10.104.106.173:2882", ack_info_list=[]) [2025-03-19 15:02:40.321540] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="10.104.106.173:2882", tmp_server={server:"10.104.106.173:2882", timestamp:1, flag:0}) [2025-03-19 15:02:40.321558] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17] current log_task status(palf_id=1002, self="10.104.106.173:2882", log_id=13105901, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-03-19 15:02:40.331105] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [801890][T1004_TimerWK1][T1004][YB420A686AAD-00062F0AD654DA5E-0-0] [lt=23] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1004, current_memory_config=0) [2025-03-19 15:02:40.331110] INFO [SERVER] check_tenant_memory (ob_plugin_vector_index_scheduler.cpp:488) [801891][T1004_TimerWK2][T1004][YB420A686AAD-00062F0AD794C959-0-0] [lt=35] get vector mem limit size(ret=0, ret="OB_SUCCESS", tenant_id=1004, current_memory_config=0) [2025-03-19 15:02:40.331144] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [801891][T1004_TimerWK2][T1004][YB420A686AAD-00062F0AD794C959-0-0] [lt=27] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1004) [2025-03-19 15:02:40.331144] INFO [SERVER] execute_adapter_maintenance (ob_plugin_vector_index_scheduler.cpp:472) [801890][T1004_TimerWK1][T1004][YB420A686AAD-00062F0AD654DA5E-0-0] [lt=34] finish generate tenant tablet tasks(ret=0, ret="OB_SUCCESS", tenant_id=1004) [2025-03-19 15:02:40.331554] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [785068][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=21] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):61440, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):61440, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):364, sequence:4}) [2025-03-19 15:02:40.341634] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [784638][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=19] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1742195121233216, tenant_id:1004, sequence_id:2199023260869}) [2025-03-19 15:02:40.341667] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [784638][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=32] try reload schema success(schema_version=1741079476939168, refresh_schema_info={schema_version:1742195121233216, tenant_id:1004, sequence_id:2199023260869}, schema_ret=0) [2025-03-19 15:02:40.341664] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [784729][SerScheQueue1][T0][YB420A686AAD-00062F071E27F1CA-0-0] [lt=20] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1742195121233216, tenant_id:1004, sequence_id:2199023260869}) [2025-03-19 15:02:40.341712] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [784729][SerScheQueue1][T0][YB420A686AAD-00062F071E27F1CA-0-0] [lt=45] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1742195121233216, tenant_id:1004, sequence_id:2199023260869}, schema_info={schema_version:1742195121233216, tenant_id:1004, sequence_id:2199023260869}) [2025-03-19 15:02:40.341722] WDIAG [SERVER.OMT] got_version (ob_tenant_config_mgr.cpp:625) [784638][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4201] No tenant config found(tenant_id=1001, ret=-4201) [2025-03-19 15:02:40.341741] WDIAG [SERVER.OMT] got_versions (ob_tenant_config_mgr.cpp:613) [784638][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4201] fail got version(tenant_id=1001, version=2, ret=-4201) [2025-03-19 15:02:40.341751] WDIAG [SERVER.OMT] got_version (ob_tenant_config_mgr.cpp:625) [784638][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4201] No tenant config found(tenant_id=1002, ret=-4201) [2025-03-19 15:02:40.341764] WDIAG [SERVER.OMT] got_versions (ob_tenant_config_mgr.cpp:613) [784638][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4201] fail got version(tenant_id=1002, version=1740712322874224, ret=-4201) [2025-03-19 15:02:40.349849] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [2924119][T1004_L0_G1][T1004][YB420A686AAF-00062F075C3F3B5D-0-0] [lt=28] receive register_req from acceptor, ignore(self="10.104.106.173:2882", palf_id=1, child={server:"10.104.106.175:2882", region:"sys_region", register_time_us:1742367760349405, keepalive_ts:-1}) [2025-03-19 15:02:40.349888] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [2924119][T1004_L0_G1][T1004][YB420A686AAF-00062F075C3F3B5D-0-0] [lt=37] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.104.106.173:2882", child={server:"10.104.106.175:2882", region:"sys_region", register_time_us:1742367760349405, keepalive_ts:-1}, is_to_leader=true) [2025-03-19 15:02:40.358875] INFO [PALF] read (log_cache.cpp:103) [802142][T1003_LogShared][T1003][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=12222, hit_cnt=17, read_cnt=17, hit rate=1.000000000000000000e+00) [2025-03-19 15:02:40.376570] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [801848][T1004_TenantInf][T1004][Y0-0000000000000000-0-0] [lt=34][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-03-19 15:02:40.376608] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [801848][T1004_TenantInf][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-03-19 15:02:40.378777] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [801848][T1004_TenantInf][T1003][YB420A686AAD-00062F0AD8FD5EE0-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-03-19 15:02:40.383493] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [784639][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=4) [2025-03-19 15:02:40.386905] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [801900][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=18] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.104.106.173:2882", has_set_deleted:false}, accum_size=1342) [2025-03-19 15:02:40.387129] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [801901][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"10.104.106.173:2882", has_set_deleted:false}, accum_size=1342) [2025-03-19 15:02:40.387186] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [801899][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=32] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="10.104.106.173:2882", committed size=1220) [2025-03-19 15:02:40.387246] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [2924119][T1004_L0_G1][T1004][YB420A686AAE-00062F08B5415EBC-0-0] [lt=24] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.104.106.173:2882", committed size=1342) [2025-03-19 15:02:40.387655] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [785116][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-03-19 15:02:40.387656] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [802173][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-03-19 15:02:40.389154] INFO [LIB] stat (utility.h:1210) [801945][T1004_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=52] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=42, this=0x7eff09ff3470, extra_info="") [2025-03-19 15:02:40.389206] INFO [LIB] stat (utility.h:1210) [801945][T1004_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=38] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x7eff09fdc050, extra_info="id:1") [2025-03-19 15:02:40.389395] INFO [LIB] stat (utility.h:1210) [802140][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=32] [PALF STAT WRITE LOG COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=208, this=0x7eff09ffdc30, extra_info="palf_id:1") [2025-03-19 15:02:40.389439] INFO [LIB] stat (utility.h:1227) [802140][T1003_IOWorker][T1003][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=19, this=0x7eff1a7ea530, extra_info="") [2025-03-19 15:02:40.389462] INFO [LIB] stat (utility.h:1210) [802139][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=51] [PALF STAT FLUSH CB COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=10, this=0x7eff09ffde60, extra_info="palf_id:1") [2025-03-19 15:02:40.389744] INFO [LIB] stat (utility.h:1210) [2924099][T1003_L0_G1][T1003][YB420A686AAF-00062F088448A2C9-0-0] [lt=35] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=10, this=0x7eff09ff36a0, extra_info="palf_id:1") [2025-03-19 15:02:40.389774] INFO [LIB] stat (utility.h:1210) [2924099][T1003_L0_G1][T1003][YB420A686AAF-00062F088448A2C9-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=624, this=0x7eff09ff38d0, extra_info="palf_id:1") [2025-03-19 15:02:40.392749] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [785109][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1, current_version={val:1742367760190966000, v:0}, min_version={val:1742367760190966000, v:0}, max_version={val:1742367760190966000, v:0}, in_service=true, leader_epoch=2) [2025-03-19 15:02:40.395620] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [2924099][T1003_L0_G1][T1003][YB420A686AAE-00062F075C05F39F-0-0] [lt=17] receive register_req from acceptor, ignore(self="10.104.106.173:2882", palf_id=1, child={server:"10.104.106.174:2882", region:"sys_region", register_time_us:1742367760395734, keepalive_ts:-1}) [2025-03-19 15:02:40.395652] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [2924099][T1003_L0_G1][T1003][YB420A686AAE-00062F075C05F39F-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.104.106.173:2882", child={server:"10.104.106.174:2882", region:"sys_region", register_time_us:1742367760395734, keepalive_ts:-1}, is_to_leader=true) [2025-03-19 15:02:40.406894] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [784636][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-394047483084, global_cache_size=21694087168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1003, 1004]) [2025-03-19 15:02:40.411809] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [785038][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=30] ====== tenant freeze timer task ====== [2025-03-19 15:02:40.412008] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:40.421298] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=31] [PALF_DUMP](palf_id=1003, self="10.104.106.173:2882", [SlidingWindow]={palf_id:1003, self:"10.104.106.173:2882", lsn_allocator:{max_log_id:12958380, max_lsn:1579122215, max_scn:{val:1742367760013812004, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:1579122215}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:1537179175}}, last_submit_lsn:{lsn:1579122093}, last_submit_end_lsn:{lsn:1579122215}, last_submit_log_id:12958380, last_submit_log_pid:1, max_flushed_lsn:{lsn:1579122093}, max_flushed_end_lsn:{lsn:1579122215}, max_flushed_log_pid:1, committed_end_lsn:{lsn:1579122215}, last_slide_log_id:12958380, last_slide_scn:{val:1742367760013812004, v:0}, last_slide_lsn:{lsn:1579122093}, last_slide_end_lsn:{lsn:1579122215}, last_slide_log_pid:1, last_slide_log_accum_checksum:1949444034, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7efdb10d2530}, [StateMgr]={this:0x7efdb10d6830, self:"10.104.106.173:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"10.104.106.175:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:12958381, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x7efdb10dd2f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"10.104.106.173:2882", alive_paxos_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:1, config_seq:7}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:1, lock_owner:10, lock_time:1741317368444270}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:8}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:10, lock_time:1741317368717644}}, prev_log_proposal_id:1, prev_lsn:{lsn:292215207}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:8}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:1742367759320679, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1742367759320679, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7efdb10d3c70}, [ModeMgr]={palf_id:1003, self:"10.104.106.173:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476772834000, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476772834000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476772834000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:1741079476772834001, v:0}, base_lsn_for_block_gc:{lsn:1476304896}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:1, config_seq:7}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:1, lock_owner:10, lock_time:1741317368444270}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:8}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:10, lock_time:1741317368717644}}, prev_log_proposal_id:1, prev_lsn:{lsn:292215207}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:1476304896}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1741079476772834000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1741079476772834000, v:0}}}, log_meta_storage:{log_tail:{lsn:118784}, readable_log_tail:{lsn:118784}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1610514432}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1004/1003/meta", dir_fd:428, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66985984, block_header_serialize_buf_:0x7efdb10d8a38, flashback_version:0}, log_storage:{log_tail:{lsn:1579122215}, readable_log_tail:{lsn:1579122215}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:1543409664}, min_scn:{val:1742338590423429005, v:0}, curr_block_id:23, palf_id:1003}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1004/1003/log", dir_fd:429, min_block_id:0, max_block_id:24, curr_writable_block_id:23}, logical_block_size_:67104768, curr_block_writable_size_:31392217, block_header_serialize_buf_:0x7efdb10da738, flashback_version:0}, palf_epoch:4, last_purge_throttling_ts:-1, this:0x7efdb10d6f70}, [Reconfirm]={palf_id:1003, self:"10.104.106.173:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7efdb10d69b0}) [2025-03-19 15:02:40.421579] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=276][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="10.104.106.173:2882", tmp_server={server:"10.104.106.173:2882", timestamp:1, flag:0}) [2025-03-19 15:02:40.421597] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17] [PALF_DUMP](palf_id=1003, self="10.104.106.173:2882", ack_info_list=[]) [2025-03-19 15:02:40.421610] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2773) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1003, self="10.104.106.173:2882", tmp_server={server:"10.104.106.173:2882", timestamp:1, flag:0}) [2025-03-19 15:02:40.421627] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [801907][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16] current log_task status(palf_id=1003, self="10.104.106.173:2882", log_id=12958381, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-03-19 15:02:40.487578] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [2924074][T1004_L0_G1][T1004][YB420A686AAF-00062F088EE0DD94-0-0] [lt=16] [PALF STAT LOG TASK TIME](palf_id=1001, self="10.104.106.173:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=2, avg_log_gen_to_submit_time=41, avg_log_submit_to_flush_time=325, avg_log_submit_to_slide_time=702) [2025-03-19 15:02:40.487634] INFO [PALF] read (log_cache.cpp:103) [801905][T1004_LogShared][T1004][Y0-0000000000000000-0-0] [lt=38] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-03-19 15:02:40.496996] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [784854][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1742367760496963], gts:1742367760496758000, latest_srr:[mts=1742367760496963]}) [2025-03-19 15:02:40.497038] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [784854][TsMgr][T0][Y0-0000000000000000-0-0] [lt=41] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-03-19 15:02:40.497076] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [784854][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts statistics(tenant_id=1003, gts_rpc_cnt=10, get_gts_cache_cnt=572, get_gts_with_stc_cnt=83, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=79, try_wait_gts_elapse_cnt=0) [2025-03-19 15:02:40.497097] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [784854][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1742367760497053], gts:1742367760496758000, latest_srr:[mts=1742367760497053]}) [2025-03-19 15:02:40.497512] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [784854][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts statistics(tenant_id=1004, gts_rpc_cnt=10, get_gts_cache_cnt=487, get_gts_with_stc_cnt=3, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=3, try_wait_gts_elapse_cnt=0) [2025-03-19 15:02:40.497531] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [784854][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1742367760497483], gts:1742367760496758000, latest_srr:[mts=1742367760497483]}) [2025-03-19 15:02:40.512068] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:40.514564] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [98008][T1_L0_G5][T1][YB420A686AAF-00062F0885830F82-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1742367760514547000, start_id=1742367760514547000, end_id=1742367760514547001) [2025-03-19 15:02:40.518903] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [801902][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=16] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"10.104.106.173:2882", has_set_deleted:false}, accum_size=1211) [2025-03-19 15:02:40.519239] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [2924119][T1004_L0_G1][T1004][YB420A686AAF-00062F088EE0DD95-0-0] [lt=31] [PALF STAT LOG TASK TIME](palf_id=1002, self="10.104.106.173:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=10, avg_log_submit_to_flush_time=235, avg_log_submit_to_slide_time=566) [2025-03-19 15:02:40.519247] INFO [PALF] read (log_cache.cpp:103) [801979][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=35] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-03-19 15:02:40.519285] INFO [PALF] next (palf_iterator.h:183) [801979][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=34] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:13102270, miss_cnt:0, cache_read_size:11925308890, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-03-19 15:02:40.519411] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [2924119][T1004_L0_G1][T1004][YB420A686AAF-00062F0886284171-0-0] [lt=33] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="10.104.106.173:2882", committed size=1220) [2025-03-19 15:02:40.526404] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [801955][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=33] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1003}, {id:1002}, {id:1}]) [2025-03-19 15:02:40.532483] INFO eloop_run (eloop.c:145) [784715][pnio2][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1742367760532481, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-03-19 15:02:40.546155] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [785027][T1_COMMONLSSe][T1][YB420A686AAD-00062F0999ED92E9-0-0] [lt=27][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-03-19 15:02:40.549588] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [2924003][T1_L0_G1][T1][YB420A686AAF-00062F071C6779B8-0-0] [lt=16] receive register_req from acceptor, ignore(self="10.104.106.173:2882", palf_id=1, child={server:"10.104.106.175:2882", region:"sys_region", register_time_us:1742367760549176, keepalive_ts:-1}) [2025-03-19 15:02:40.549621] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [2924003][T1_L0_G1][T1][YB420A686AAF-00062F071C6779B8-0-0] [lt=32] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="10.104.106.173:2882", child={server:"10.104.106.175:2882", region:"sys_region", register_time_us:1742367760549176, keepalive_ts:-1}, is_to_leader=true) [2025-03-19 15:02:40.553362] INFO do_work (ob_rl_mgr.cpp:710) [784727][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=33] swc wakeup.(stat_period_=1000000, ready=false) [2025-03-19 15:02:40.559155] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [802166][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1003, new_min_version={val:1742367760357485000, v:0}, new_max_version={val:1742367760357485000, v:0}, last_min_version={val:1742367760158259000, v:0}, last_max_version={val:1742367760158259000, v:0}, total_time_us=2548, delta_us=201668) [2025-03-19 15:02:40.583595] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [784639][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=3) [2025-03-19 15:02:40.592313] INFO [LIB] stat (utility.h:1210) [785078][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=178, this=0x7effbcbcdc30, extra_info="") [2025-03-19 15:02:40.592346] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [785078][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=32] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"10.104.106.173:2882", has_set_deleted:false}, accum_size=3544) [2025-03-19 15:02:40.592366] INFO [LIB] stat (utility.h:1227) [785078][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=15, this=0x7effcf9f8d30, extra_info="") [2025-03-19 15:02:40.592379] INFO [LIB] stat (utility.h:1210) [785077][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x7effbcbcde60, extra_info="") [2025-03-19 15:02:40.607248] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [784636][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=38] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-394047483084, global_cache_size=21694087168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1003, 1004]) [2025-03-19 15:02:40.611528] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=13] ====== checkpoint timer task ====== [2025-03-19 15:02:40.612129] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:40.612553] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=27] succ to get rec scn(*this={this:0x7efd39604030, ls_id:{id:1}, tenant_id:1004, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1741079458924982, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-03-19 15:02:40.612620] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=65] tx ctx memtable get rec scn(this={ObITable:{this:0x7efe9d9e2080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7efe9d9e2080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-03-19 15:02:40.612928] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=36] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1742320905636171002, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:60379}, rec_scn:{val:1742320905636171002, v:0}}, }, *this={this:0x7efdd2dc3150, is_inited:true, freezing_scn:{val:1742320803578649000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1004, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1742320804385836000, v:0}, clog_base_lsn:{lsn:2080247808}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1742367757510035001, v:0}, tablet_change_checkpoint_scn:{val:1742124392298299003, v:0}, all_id_meta:{id_meta:[{limited_id:1742367774485336000, latest_log_ts:{val:1742367757972812001, v:0}}, {limited_id:7000001, latest_log_ts:{val:1742291239472818005, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x7efd20df8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-03-19 15:02:40.613044] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=112] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7efdd2dc21b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7efdd2dcd010, memtable_head:24, memtable_tail:25, t3m:0x7efcdc8ea030, tables:[null, null, null, null, null, null, null, null, 0x7efe9d9d7bf0, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:1742320804221784, tx_data_table:0x7efdd2dced10, ls_tablet_svr:0x7efdd2dc2190}, min_rec_scn={val:1742320804385836000, v:0}, ls_id_={id:1}) [2025-03-19 15:02:40.613089] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=36] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-03-19 15:02:40.613111] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=19] get rec log scn(service_type_=0, rec_log_ts={val:1742367757972812001, v:0}) [2025-03-19 15:02:40.613127] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-03-19 15:02:40.613141] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=7] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1741319013642773001, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1741319013642773001, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-03-19 15:02:40.613168] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=21] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1741319013642773001, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1741319013642773001, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-03-19 15:02:40.613188] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1742320804385836000, v:0}, checkpoint_scn_in_ls_meta={val:1742320804385836000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-03-19 15:02:40.613218] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"10.104.106.173:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2080247808}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2080247808}, lsn={lsn:2080247808}) [2025-03-19 15:02:40.613248] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=29] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2080247808}, ls->get_ls_id()={id:1}) [2025-03-19 15:02:40.614249] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=15] succ to get rec scn(*this={this:0x7efd396b6030, ls_id:{id:1001}, tenant_id:1004, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1741079476753765001, v:0}, max_applied_start_working_ts_:{val:1741079476753765001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1741079485411835, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:196}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-03-19 15:02:40.614290] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=40] tx ctx memtable get rec scn(this={ObITable:{this:0x7efe9d9e3350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7efe9d9e3350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-03-19 15:02:40.614389] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=26] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1742320906638205004, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606847525}, rec_scn:{val:1742320906638205004, v:0}}, }, *this={this:0x7efd7e92f150, is_inited:true, freezing_scn:{val:1742320803578649000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1004, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1742320906638205004, v:0}, clog_base_lsn:{lsn:1476304896}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1742367757510035001, v:0}, tablet_change_checkpoint_scn:{val:1742124392298299004, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1741317367430813003, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:17180067072(4, 3, 5, 0)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x7efd20df8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-03-19 15:02:40.614483] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=92] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x7efd7e92f150, is_inited:true, freezing_scn:{val:1742320803578649000, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1004, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1742320906638205004, v:0}, clog_base_lsn:{lsn:1476304896}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1742367757510035001, v:0}, tablet_change_checkpoint_scn:{val:1742124392298299004, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1741317367430813003, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:17180067072(4, 3, 5, 0)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x7efd20df8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1742320906638205004, v:0}, ls_id_={id:1001}) [2025-03-19 15:02:40.614570] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=82] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1741079476753765000, v:0}) [2025-03-19 15:02:40.614595] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=22] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1741319013642773000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1741319013642773000, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-03-19 15:02:40.614618] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=20] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1741319013642773000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1741319013642773000, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-03-19 15:02:40.614636] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1742320906638205004, v:0}, checkpoint_scn_in_ls_meta={val:1742320906638205004, v:0}, ls_id={id:1001}, service_type="TRANS_SERVICE") [2025-03-19 15:02:40.614657] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"10.104.106.173:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:1476304896}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1741079476753765000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:1476304896}, lsn={lsn:1476304896}) [2025-03-19 15:02:40.614682] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:1476304896}, ls->get_ls_id()={id:1001}) [2025-03-19 15:02:40.615776] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=34] succ to get rec scn(*this={this:0x7efee4e04030, ls_id:{id:1002}, tenant_id:1004, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1741079476766497001, v:0}, max_applied_start_working_ts_:{val:1741079476766497001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1741079485512444, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:193}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-03-19 15:02:40.615819] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=43] tx ctx memtable get rec scn(this={ObITable:{this:0x7efe9d9e4620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7efe9d9e4620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-03-19 15:02:40.615919] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=26] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1742320919260989005, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606847616}, rec_scn:{val:1742320919260989005, v:0}}, }, *this={this:0x7efcb8817150, is_inited:true, freezing_scn:{val:1742320803575475001, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1004, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1742320919260989005, v:0}, clog_base_lsn:{lsn:11810439168}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1742367757510035001, v:0}, tablet_change_checkpoint_scn:{val:1742124392475704002, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1741317364518548002, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:17180067072(4, 3, 5, 0)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x7efd20df8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-03-19 15:02:40.616000] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=79] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x7efcb8817150, is_inited:true, freezing_scn:{val:1742320803575475001, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1004, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1742320919260989005, v:0}, clog_base_lsn:{lsn:11810439168}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1742367757510035001, v:0}, tablet_change_checkpoint_scn:{val:1742124392475704002, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1741317364518548002, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:17180067072(4, 3, 5, 0)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x7efd20df8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1742320919260989005, v:0}, ls_id_={id:1002}) [2025-03-19 15:02:40.616069] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=66] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1741079476766497000, v:0}) [2025-03-19 15:02:40.616088] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=17] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1741316630557356000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1741316630557356000, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-03-19 15:02:40.616109] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=19] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1741316630557356000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1741316630557356000, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-03-19 15:02:40.616128] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1742320919260989005, v:0}, checkpoint_scn_in_ls_meta={val:1742320919260989005, v:0}, ls_id={id:1002}, service_type="TRANS_SERVICE") [2025-03-19 15:02:40.616149] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"10.104.106.173:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:11810439168}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1741079476766497000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:11810439168}, lsn={lsn:11810439168}) [2025-03-19 15:02:40.616174] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:11810439168}, ls->get_ls_id()={id:1002}) [2025-03-19 15:02:40.617192] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=14] succ to get rec scn(*this={this:0x7efee4eb6030, ls_id:{id:1003}, tenant_id:1004, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1741079476772834001, v:0}, max_applied_start_working_ts_:{val:1741079476772834001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1741079485604867, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:176}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-03-19 15:02:40.617231] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=40] tx ctx memtable get rec scn(this={ObITable:{this:0x7efe9d9e58f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7efe9d9e58f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-03-19 15:02:40.617326] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=26] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1742320889868941002, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:200426}, rec_scn:{val:1742320889868941002, v:0}}, }, *this={this:0x7efdb1067150, is_inited:true, freezing_scn:{val:1742320803233582004, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1004, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1742320889868941002, v:0}, clog_base_lsn:{lsn:1476304896}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1742367757510035001, v:0}, tablet_change_checkpoint_scn:{val:1742124392352149003, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1741317368424038001, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:17180067072(4, 3, 5, 0)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x7efd20df8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-03-19 15:02:40.617406] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=79] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x7efdb1067150, is_inited:true, freezing_scn:{val:1742320803233582004, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1004, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1742320889868941002, v:0}, clog_base_lsn:{lsn:1476304896}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1742367757510035001, v:0}, tablet_change_checkpoint_scn:{val:1742124392352149003, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1741317368424038001, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:17180067072(4, 3, 5, 0)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x7efd20df8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1742320889868941002, v:0}, ls_id_={id:1003}) [2025-03-19 15:02:40.617491] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=82] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1741079476772834000, v:0}) [2025-03-19 15:02:40.617512] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=18] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1741267724343560000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1741267724343560000, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-03-19 15:02:40.617533] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=20] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1741267724343560000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:1741267724343560000, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-03-19 15:02:40.617551] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1742320889868941002, v:0}, checkpoint_scn_in_ls_meta={val:1742320889868941002, v:0}, ls_id={id:1003}, service_type="TRANS_SERVICE") [2025-03-19 15:02:40.617572] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"10.104.106.173:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:1476304896}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1741079476772834000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:1476304896}, lsn={lsn:1476304896}) [2025-03-19 15:02:40.617597] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:1476304896}, ls->get_ls_id()={id:1003}) [2025-03-19 15:02:40.617618] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=19] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-03-19 15:02:40.634567] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [785069][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=29] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:932, free_obj_hold_cnt:152, allocator used:4171232, allocator total:4439040}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:146, free_obj_hold_cnt:938, allocator used:71100644, allocator total:71100644}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=75539684, tablets_mem_limit=6888530480, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:40, free_obj_hold_cnt:91, allocator used:268288, allocator total:326400}, pending_cnt=0, wait_gc_count=0, tablet count=1078) [2025-03-19 15:02:40.659583] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [802144][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=28] [PALF_DUMP](palf_id=1, self="10.104.106.173:2882", [SlidingWindow]={palf_id:1, self:"10.104.106.173:2882", lsn_allocator:{max_log_id:21664872, max_lsn:13890797131, max_scn:{val:1742367760588234000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:13890797131}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:13848854091}}, last_submit_lsn:{lsn:13890796542}, last_submit_end_lsn:{lsn:13890797131}, last_submit_log_id:21664872, last_submit_log_pid:1, max_flushed_lsn:{lsn:13890796542}, max_flushed_end_lsn:{lsn:13890797131}, max_flushed_log_pid:1, committed_end_lsn:{lsn:13890797131}, last_slide_log_id:21664872, last_slide_scn:{val:1742367760588234000, v:0}, last_slide_lsn:{lsn:13890796542}, last_slide_end_lsn:{lsn:13890797131}, last_slide_log_pid:1, last_slide_log_accum_checksum:4286228024, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7eff09ff2530}, [StateMgr]={this:0x7eff09ff6830, self:"10.104.106.173:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"10.104.106.173:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:21664873, is_changing_config_with_arb:false, reconfirm_start_time_us:1741079443971438, palf_role_change_cb:0x7eff09ffd2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"10.104.106.173:2882", alive_paxos_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7eff09ff3c70}, [ModeMgr]={palf_id:1, self:"10.104.106.173:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1741079444171649, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1741434694194244001, v:0}, min_block_id:55, min_block_min_scn:{val:1741428262172390001, v:0}, base_lsn_for_block_gc:{lsn:13353848832}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"10.104.106.173:2882", timestamp:1, flag:0}{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:13353848832}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:118784}, readable_log_tail:{lsn:118784}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:13957791744}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1003/1/meta", dir_fd:400, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66985984, block_header_serialize_buf_:0x7eff09ff8a38, flashback_version:0}, log_storage:{log_tail:{lsn:13890797131}, readable_log_tail:{lsn:13890797131}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:13890686976}, min_scn:{val:1742367750577789001, v:0}, curr_block_id:207, palf_id:1}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1003/1/log", dir_fd:403, min_block_id:55, max_block_id:208, curr_writable_block_id:207}, logical_block_size_:67104768, curr_block_writable_size_:66994613, block_header_serialize_buf_:0x7eff09ffa738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1741079443971346, this:0x7eff09ff6f70}, [Reconfirm]={palf_id:1, self:"10.104.106.173:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"10.104.106.174:2882";"10.104.106.175:2882";, curr_paxos_follower_list:2{server:"10.104.106.174:2882", timestamp:1, flag:0}{server:"10.104.106.175:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"10.104.106.174:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1741079443972073, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7eff09ff69b0}) [2025-03-19 15:02:40.659916] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [802144][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=327] [PALF_DUMP](palf_id=1, self="10.104.106.173:2882", ack_info_list=[{member:{server:"10.104.106.173:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760589387, last_flushed_end_lsn:{lsn:13890797131}}, {member:{server:"10.104.106.174:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760589649, last_flushed_end_lsn:{lsn:13890797131}}, {member:{server:"10.104.106.175:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760589776, last_flushed_end_lsn:{lsn:13890797131}}]) [2025-03-19 15:02:40.659945] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [802144][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=26] current log_task status(palf_id=1, self="10.104.106.173:2882", log_id=21664873, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"10.104.106.173:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760589387, last_flushed_end_lsn:{lsn:13890797131}}, {member:{server:"10.104.106.174:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760589649, last_flushed_end_lsn:{lsn:13890797131}}, {member:{server:"10.104.106.175:2882", timestamp:1, flag:0}, last_ack_time_us:1742367760589776, last_flushed_end_lsn:{lsn:13890797131}}]) [2025-03-19 15:02:40.660247] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [802129][T1003_TimerWK0][T1003][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1003, evict_high_mem_level=103079214, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=95668224) [2025-03-19 15:02:40.660302] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [802131][T1003_TimerWK2][T1003][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=1659, get_cur_idx()=1651, get_pop_idx()=1651) [2025-03-19 15:02:40.676103] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [801918][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=23] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=25, log_id=5361050, offset=10389237}, end_cursor:ObLogCursor{file_id=25, log_id=5361051, offset=10389543}, is_inited:true, is_local:false, buf_size:8192, buf:0x7f003b2a4050, len:2315, log_data_len:306, seq:5361050, flush_finish:false, flush_ret:0}) [2025-03-19 15:02:40.676335] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [2922957][T1004_L0_G18][T1004][YB420A686AAD-00062F0A7DD577C5-0-0] [lt=33] start get_ls_replayed_scn(arg={tenant_id:1004, ls_id:{id:1}, all_replica:false}) [2025-03-19 15:02:40.676380] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [2922957][T1004_L0_G18][T1004][YB420A686AAD-00062F0A7DD577C5-0-0] [lt=34] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1742367760496758000, v:0}, arg={tenant_id:1004, ls_id:{id:1}, all_replica:false}, result={tenant_id:1004, ls_id:{id:1}, cur_readable_scn:{val:1742367760496758000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"10.104.106.173:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-03-19 15:02:40.677240] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [801892][T1004_TimerWK3][T1004][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1004, evict_high_mem_level=927712935, evict_high_size_level=884736, evict_batch_count=0, elapse_time=0, size_used=458752, mem_used=900529152) [2025-03-19 15:02:40.677306] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [801891][T1004_TimerWK2][T1004][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=974, get_cur_idx()=966, get_pop_idx()=959) [2025-03-19 15:02:40.677776] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [2922957][T1004_L0_G18][T1004][YB420A686AAF-00062F085264BF6A-0-0] [lt=32] start get_ls_replayed_scn(arg={tenant_id:1004, ls_id:{id:1003}, all_replica:false}) [2025-03-19 15:02:40.677818] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [2922957][T1004_L0_G18][T1004][YB420A686AAF-00062F085264BF6A-0-0] [lt=30] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1742367760514547001, v:0}, arg={tenant_id:1004, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1004, ls_id:{id:1003}, cur_readable_scn:{val:1742367760514547001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"10.104.106.173:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-03-19 15:02:40.678932] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [785068][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=42] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-03-19 15:02:40.678971] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [785068][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=28] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=4620) [2025-03-19 15:02:40.680621] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [801851][T1004_TeRec][T1003][YB420A686AAD-00062F0A7DD577C5-0-0] [lt=33][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-03-19 15:02:40.682266] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [801918][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=46] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=25, log_id=5361051, offset=10389543}, end_cursor:ObLogCursor{file_id=25, log_id=5361052, offset=10389852}, is_inited:true, is_local:false, buf_size:8192, buf:0x7f003b2c0050, len:2009, log_data_len:309, seq:5361051, flush_finish:false, flush_ret:0}) [2025-03-19 15:02:40.682490] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [2922957][T1004_L0_G18][T1004][YB420A686AAD-00062F0A7DD577C5-0-0] [lt=38] start get_ls_replayed_scn(arg={tenant_id:1004, ls_id:{id:1001}, all_replica:false}) [2025-03-19 15:02:40.682523] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [2922957][T1004_L0_G18][T1004][YB420A686AAD-00062F0A7DD577C5-0-0] [lt=24] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1742367760496758000, v:0}, arg={tenant_id:1004, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1004, ls_id:{id:1001}, cur_readable_scn:{val:1742367760496758000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"10.104.106.173:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-03-19 15:02:40.685122] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [802145][T1003_ApplySrv0][T1003][Y0-0000000000000000-0-0] [lt=26] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-03-19 15:02:40.686095] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [801918][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=35] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=25, log_id=5361052, offset=10389852}, end_cursor:ObLogCursor{file_id=25, log_id=5361053, offset=10390161}, is_inited:true, is_local:false, buf_size:8192, buf:0x7f003b270050, len:1700, log_data_len:309, seq:5361052, flush_finish:false, flush_ret:0}) [2025-03-19 15:02:40.686310] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [801918][T1004_OB_SLOG][T1004][Y0-0000000000000000-0-0] [lt=31] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=25, log_id=5361053, offset=10390161}, end_cursor:ObLogCursor{file_id=25, log_id=5361054, offset=10390470}, is_inited:true, is_local:false, buf_size:8192, buf:0x7f003b2d4050, len:1391, log_data_len:309, seq:5361053, flush_finish:false, flush_ret:0}) [2025-03-19 15:02:40.712191] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:40.712712] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [785070][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=38] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=826623659, evict_high_size_level=825753, evict_batch_count=0, elapse_time=0, size_used=393216, mem_used=725830656) [2025-03-19 15:02:40.712766] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [785069][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=56] Construct Queue Num(construct_num=0, get_push_idx()=2267, get_cur_idx()=2259, get_pop_idx()=2253) [2025-03-19 15:02:40.718703] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAE-00062F08B1848CAC-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x7efcf40b9470, extra_info="") [2025-03-19 15:02:40.718745] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [2924119][T1004_L0_G1][T1004][YB420A686AAE-00062F08B1848CAC-0-0] [lt=31] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="10.104.106.173:2882", committed size=1220) [2025-03-19 15:02:40.718947] INFO [LIB] stat (utility.h:1210) [801902][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=38] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=189, this=0x7efcf40c3c30, extra_info="palf_id:1002") [2025-03-19 15:02:40.718992] INFO [LIB] stat (utility.h:1227) [801902][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=35] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x7efd20dff3b0, extra_info="") [2025-03-19 15:02:40.719041] INFO [LIB] stat (utility.h:1210) [801899][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=28] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=36, this=0x7efcf40c3e60, extra_info="palf_id:1002") [2025-03-19 15:02:40.719422] INFO [LIB] stat (utility.h:1210) [801973][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=127] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=22, this=0x7efda688a8f8, extra_info="") [2025-03-19 15:02:40.719455] INFO [LIB] stat (utility.h:1210) [801973][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=32] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=3, this=0x7efda688ab28, extra_info="") [2025-03-19 15:02:40.723547] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [785092][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=37] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-03-19 15:02:40.723546] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [801919][T1004_TFSwap][T1004][Y0-0000000000000000-0-0] [lt=37] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-03-19 15:02:40.723584] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [785092][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-03-19 15:02:40.723584] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [801919][T1004_TFSwap][T1004][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-03-19 15:02:40.723602] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [801919][T1004_TFSwap][T1004][Y0-0000000000000000-0-0] [lt=16] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-03-19 15:02:40.723602] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [785092][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-03-19 15:02:40.723626] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [785092][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=42251, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-03-19 15:02:40.723626] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [801919][T1004_TFSwap][T1004][Y0-0000000000000000-0-0] [lt=21] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47311, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-03-19 15:02:40.723647] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [801919][T1004_TFSwap][T1004][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:2248749039091, flush_io_finished_ret_:0, flush_io_finished_round_:1288099, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-03-19 15:02:40.723647] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [785092][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:2248749039091, flush_io_finished_ret_:0, flush_io_finished_round_:1810032, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-03-19 15:02:40.783713] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [784639][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=2) [2025-03-19 15:02:40.785664] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [785110][T1_LSArchiveMgr][T1][YB420A686AAD-00062F071A9100C5-0-0] [lt=28] gc stale ls task succ [2025-03-19 15:02:40.786923] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=18] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.104.106.173:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-03-19 15:02:40.786959] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=33] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.104.106.173:2882", has_set_deleted:false}, append size=1220) [2025-03-19 15:02:40.789622] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [802100][T1003_COMMONLSS][T1][YB420A686AAD-00062F0A7DF57411-0-0] [lt=60][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-03-19 15:02:40.791240] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [802100][T1003_COMMONLSS][T1003][YB420A686AAD-00062F0A7DF57411-0-0] [lt=32][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-03-19 15:02:40.806682] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [801841][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=47] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-03-19 15:02:40.807711] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [784636][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-394047483084, global_cache_size=21694087168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1003, 1004]) [2025-03-19 15:02:40.812258] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:40.814862] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [784637][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=33] skip inspect bad block(last_check_time=1742287268941776, last_macro_idx=-1) [2025-03-19 15:02:40.819519] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAF-00062F0886284177-0-0] [lt=21] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x7efdb10d3470, extra_info="") [2025-03-19 15:02:40.819761] INFO [LIB] stat (utility.h:1210) [801903][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=12] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=192, this=0x7efdb10ddc30, extra_info="palf_id:1003") [2025-03-19 15:02:40.819794] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [801903][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=32] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"10.104.106.173:2882", has_set_deleted:false}, accum_size=1220) [2025-03-19 15:02:40.819816] INFO [LIB] stat (utility.h:1227) [801903][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x7efd20dffaf0, extra_info="") [2025-03-19 15:02:40.819857] INFO [LIB] stat (utility.h:1210) [801899][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=25, this=0x7efdb10dde60, extra_info="palf_id:1003") [2025-03-19 15:02:40.830002] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [802189][T1003_L5_G0][T1003][YB420A686AAF-00062F084EBD86BD-0-0] [lt=37][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-03-19 15:02:40.842163] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784860][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.842197] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784864][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.842241] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784857][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.842248] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784856][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.842723] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784858][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.842726] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784859][RpcIO][T0][Y0-0000000000000000-0-0] [lt=29] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.842732] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784863][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.842779] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784855][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.842836] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784861][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.842997] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784867][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.842996] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784865][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.843277] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784862][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.843776] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784866][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:40.863244] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [801890][T1004_TimerWK1][T1004][Y0-0000000000000000-0-0] [lt=14] success to update min reserved snapshot(reserved_snapshot=1742365952805468000, duration=1800, snapshot_gc_ts=1742367752805468000) [2025-03-19 15:02:40.864714] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [801890][T1004_TimerWK1][T1004][Y0-0000000000000000-0-0] [lt=31] update reserved snapshot finished(cost_ts=1471, reserved_snapshot=1742365952805468000) [2025-03-19 15:02:40.891958] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [784966][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31] worker thread began to shrink(tenant_->id()=1004, group_id_=1, token=0) [2025-03-19 15:02:40.897802] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [2924074][T1004_L0_G1][T1004][Y0-0000000000000000-0-0] [lt=40] worker thread exit(tenant_->id()=1004, workers_.get_size()=2) [2025-03-19 15:02:40.911265] INFO [LIB] stat (utility.h:1210) [785117][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=42, this=0x7effbcbc3470, extra_info="") [2025-03-19 15:02:40.911308] INFO [LIB] stat (utility.h:1210) [785117][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x7effbc9e8050, extra_info="id:1") [2025-03-19 15:02:40.911507] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [785077][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.104.106.173:2882", committed size=3544) [2025-03-19 15:02:40.911855] INFO [LIB] stat (utility.h:1210) [2924003][T1_L0_G1][T1][YB420A686AAF-00062F087B6F32FC-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9, this=0x7effbcbc36a0, extra_info="palf_id:1") [2025-03-19 15:02:40.911894] INFO [LIB] stat (utility.h:1210) [2924003][T1_L0_G1][T1][YB420A686AAF-00062F087B6F32FC-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=621, this=0x7effbcbc38d0, extra_info="palf_id:1") [2025-03-19 15:02:40.911910] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [2924003][T1_L0_G1][T1][YB420A686AAF-00062F087B6F32FC-0-0] [lt=14] [PALF STAT LOG TASK TIME](palf_id=1, self="10.104.106.173:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=4, avg_log_gen_to_submit_time=58, avg_log_submit_to_flush_time=212, avg_log_submit_to_slide_time=563) [2025-03-19 15:02:40.911950] INFO [PALF] read (log_cache.cpp:103) [785080][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=37] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3544, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-03-19 15:02:40.912317] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:40.918804] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [2924119][T1004_L0_G1][T1004][YB420A686AAE-00062F08B1848CB0-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1002, self="10.104.106.173:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-03-19 15:02:40.924163] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:56) [785550][T1_L0_G10000][T1][YB420A686AAD-00062F071F5FE386-0-0] [lt=35][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=14, len=196, pos=177) [2025-03-19 15:02:40.925170] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:56) [785550][T1_L0_G10000][T1][YB420A686AAD-00062F071F5FE386-0-0] [lt=34][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=60, len=284, pos=258) [2025-03-19 15:02:40.939925] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [801837][T1004_Occam][T1004][YB420A686AAD-00062F0B05F1BB8A-0-0] [lt=21] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-03-19 15:02:40.945566] INFO eloop_run (eloop.c:145) [784702][pnio1][T0][Y0-0000000000000000-0-0] [lt=38] PNIO [ratelimit] time: 1742367760945565, bytes: 506477466185, bw: 0.251113 MB/s, add_ts: 1004439, add_bytes: 264480 [2025-03-19 15:02:40.950438] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [802189][T1003_L5_G0][T1003][YB420A686AAE-00062F085BFF4A2D-0-0] [lt=35][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-03-19 15:02:40.958520] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [802166][T1003_TenantWea][T1003][YB420A686AAD-00062F0B15C98A85-0-0] [lt=37] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=3.959010052680969238e+02, this=0x7efecde7b208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=2047, Resource Limit(MB)=1979, Throttle Trigger(MB)=1187, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=1.166419729878335421e+00) [2025-03-19 15:02:40.978642] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [785305][T1_L0_G5][T1][YB420A686AAE-00062F08A2C7E81F-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1742367760977604000, start_id=1742367760977604000, end_id=1742367760977604001) [2025-03-19 15:02:40.981278] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [801845][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=78] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1, is_allowed_broadcast=true) [2025-03-19 15:02:40.983818] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [784638][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=1) [2025-03-19 15:02:40.987093] INFO [LIB] stat (utility.h:1210) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=21] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=32, this=0x7efd7e8ff470, extra_info="") [2025-03-19 15:02:40.987132] INFO [LIB] stat (utility.h:1210) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=29] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x7efd7e938050, extra_info="id:1001") [2025-03-19 15:02:40.987396] INFO [LIB] stat (utility.h:1210) [801901][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=278, this=0x7efd7e909c30, extra_info="palf_id:1001") [2025-03-19 15:02:40.987442] INFO [LIB] stat (utility.h:1227) [801901][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=41] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x7efd20dfec70, extra_info="") [2025-03-19 15:02:40.987456] INFO [LIB] stat (utility.h:1210) [801899][T1004_LogIOCb0][T1004][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=7, this=0x7efd7e909e60, extra_info="palf_id:1001") [2025-03-19 15:02:40.987739] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAE-00062F08B5415ED1-0-0] [lt=34] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=9, this=0x7efd7e8ff6a0, extra_info="palf_id:1001") [2025-03-19 15:02:40.987762] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAE-00062F08B5415ED1-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=696, this=0x7efd7e8ff8d0, extra_info="palf_id:1001") [2025-03-19 15:02:41.008122] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [784639][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-394047483084, global_cache_size=21694087168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1003, 1004]) [2025-03-19 15:02:41.011188] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [801854][T1004_BalanceEx][T1004][YB420A686AAD-00062F0A7D55A0D0-0-0] [lt=16] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-03-19 15:02:41.011229] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [801854][T1004_BalanceEx][T1004][YB420A686AAD-00062F0A7D55A0D0-0-0] [lt=38] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-03-19 15:02:41.012378] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:41.021261] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [784639][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=39] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=20132600, clean_num=201326) [2025-03-19 15:02:41.028803] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [801946][T1004_LSArchive][T1004][YB420A686AAD-00062F086E4100C5-0-0] [lt=20] gc stale ls task succ [2025-03-19 15:02:41.028852] INFO [ARCHIVE] add_ls_task_ (ob_ls_mgr.cpp:456) [801946][T1004_LSArchive][T1004][YB420A686AAD-00062F086E4100C5-0-0] [lt=29] ls not need do archive on this server(ls_id={id:1002}) [2025-03-19 15:02:41.028874] INFO [ARCHIVE] add_ls_task_ (ob_ls_mgr.cpp:456) [801946][T1004_LSArchive][T1004][YB420A686AAD-00062F086E4100C5-0-0] [lt=20] ls not need do archive on this server(ls_id={id:1003}) [2025-03-19 15:02:41.048690] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [801979][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=42] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:11925309556}, scn={val:1742367760978659001, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-03-19 15:02:41.048660] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [801977][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=35] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:4, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1742367761048498, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:7427, ret_code_:0}}, next_to_submit_lsn_:{lsn:11925309500}, next_to_submit_scn_:{val:1742367760978659001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1741079476766497000, v:0}, iterator_:{iterator_impl:{buf_:0x7efe20f07000, next_round_pread_size:3678208, curr_read_pos:56, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:11925309500}, end_lsn:{lsn:11925309612}, read_buf:{buf_len_:638976, buf_:0x7efe20f07000}, block_size:67104768, log_storage_:0x7efcf40bfa70, log_storage_:{log_tail:{lsn:11925309612}, readable_log_tail:{lsn:11925309612}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:11877543936}, min_scn:{val:1742328730960686003, v:0}, curr_block_id:177, palf_id:1002}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1004/1002/log", dir_fd:424, min_block_id:0, max_block_id:178, curr_writable_block_id:177}, logical_block_size_:67104768, curr_block_writable_size_:19339092, block_header_serialize_buf_:0x7efcf40c0738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1742367760978659001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1742367760978659001, v:0}, data_checksum:804882841, flag:2266}}, init_mode_version:0, accumulate_checksum:431438711, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1742367760978659001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:13102276, miss_cnt:0, cache_read_size:11925309612, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:11925309556}, scn_:{val:1742367760978659001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1742367761047956, first_handle_ts_:1742367761048658, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x7ef8ea604150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-03-19 15:02:41.048952] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [801975][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=13] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1742365952805468000) [2025-03-19 15:02:41.048991] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [801975][T1004_ReplaySrv][T1004][Y0-0000000000000000-0-0] [lt=37] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:11925309556}, scn_:{val:1742367760978659001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1742367761047956, first_handle_ts_:1742367761048928, replay_cost_:61, retry_cost_:-1, read_log_buf_:0x7ef8ea604160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1742367761048498, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:7427, ret_code_:0}}, next_to_submit_lsn_:{lsn:11925309612}, next_to_submit_scn_:{val:1742367760978659002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1741079476766497000, v:0}, iterator_:{iterator_impl:{buf_:0x7efe20f07000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:11925309500}, end_lsn:{lsn:11925309612}, read_buf:{buf_len_:638976, buf_:0x7efe20f07000}, block_size:67104768, log_storage_:0x7efcf40bfa70, log_storage_:{log_tail:{lsn:11925309612}, readable_log_tail:{lsn:11925309612}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:11877543936}, min_scn:{val:1742328730960686003, v:0}, curr_block_id:177, palf_id:1002}, block_mgr:{log_dir:"/home/admin/oceanbase/store/shanxi_ob_second/clog/tenant_1004/1002/log", dir_fd:424, min_block_id:0, max_block_id:178, curr_writable_block_id:177}, logical_block_size_:67104768, curr_block_writable_size_:19339092, block_header_serialize_buf_:0x7efcf40c0738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1742367760978659001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1742367760978659001, v:0}, data_checksum:804882841, flag:2266}}, init_mode_version:0, accumulate_checksum:431438711, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1742367760978659001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:13102276, miss_cnt:0, cache_read_size:11925309612, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=20, ret=0) [2025-03-19 15:02:41.067715] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [784884][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=33] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.068134] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784875][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.068351] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784870][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.068599] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784877][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.068718] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784873][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.068723] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784876][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.068722] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784869][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.068726] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784872][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.068728] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784871][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.068865] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784874][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.076724] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784878][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.076732] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784882][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.076772] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784879][BatchIO][T0][Y0-0000000000000000-0-0] [lt=29] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.076787] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784880][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.076803] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [784881][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.077232] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [784886][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-03-19 15:02:41.084230] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [802174][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=106] tx gc loop thread is running(MTL_ID()=1003) [2025-03-19 15:02:41.084265] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [802174][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=34] try gc retain ctx [2025-03-19 15:02:41.084276] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [802174][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-03-19 15:02:41.084315] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [802174][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=7] finish update min start scn(max_decided_scn={val:1742367760997652001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1742367750990329001, v:0}, keep_alive_scn_:{val:1742367755988616000, v:0}, update_ts_:1742367761084099}) [2025-03-19 15:02:41.085379] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [802174][T1003_TxLoopWor][T1003][Y0-0000000000000000-0-0] [lt=28] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1003, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1003, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1742320803708055001, v:0}, clog_base_lsn:{lsn:13353848832}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1742118204786497000, v:0}, all_id_meta:{id_meta:[{limited_id:1742367774497129000, latest_log_ts:{val:1742367757955987001, v:0}}, {limited_id:23000001, latest_log_ts:{val:1742359877227663001, v:0}}, {limited_id:220000001, latest_log_ts:{val:1742321077556576001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x7eff1a7e4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:15831098, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-03-19 15:02:41.087171] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=19] [PALF STAT GROUP LOG INFO](palf_id=1001, self="10.104.106.173:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1332, avg_group_log_size=121) [2025-03-19 15:02:41.087204] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"10.104.106.173:2882", has_set_deleted:false}, append size=1332) [2025-03-19 15:02:41.092985] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [785068][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=14] ====== check clog disk timer task ====== [2025-03-19 15:02:41.093019] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [785068][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=30] get_disk_usage(ret=0, capacity(MB):=61440, used(MB):=10540) [2025-03-19 15:02:41.111331] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [785117][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] tx gc loop thread is running(MTL_ID()=1) [2025-03-19 15:02:41.111361] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [785117][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] try gc retain ctx [2025-03-19 15:02:41.111370] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [785117][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-03-19 15:02:41.111395] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [785117][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] finish update min start scn(max_decided_scn={val:1742367760992389000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1742367750986107001, v:0}, keep_alive_scn_:{val:1742367755988616001, v:0}, update_ts_:1742367761110508}) [2025-03-19 15:02:41.112468] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=46] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"10.104.106.173:2882", switch_to_leader_ts:-1}) [2025-03-19 15:02:41.112500] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=30] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"10.104.106.173:2882", switch_to_leader_ts:-1}) [2025-03-19 15:02:41.112519] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [801944][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-03-19 15:02:41.112465] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [785117][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1742320802173577000, v:0}, clog_base_lsn:{lsn:10669658112}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1742289476612449000, v:0}, all_id_meta:{id_meta:[{limited_id:1742367767709083000, latest_log_ts:{val:1742367751185590001, v:0}}, {limited_id:18000001, latest_log_ts:{val:1742332566823725002, v:0}}, {limited_id:290000001, latest_log_ts:{val:1742317358661621001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:2, palf_env_:0x7effcc1e4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-03-19 15:02:41.113234] INFO [LIB] stat (utility.h:1210) [96513][T1_ApplySrv2][T1][Y0-0000000000000000-0-0] [lt=37] [APPLY STAT CB APPEND COST TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=84, this=0x7effbcbf6280, extra_info="") [2025-03-19 15:02:41.113275] INFO [LIB] stat (utility.h:1210) [96513][T1_ApplySrv2][T1][Y0-0000000000000000-0-0] [lt=40] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=586, this=0x7effbcbf64b0, extra_info="") [2025-03-19 15:02:41.113299] INFO [LIB] stat (utility.h:1210) [96513][T1_ApplySrv2][T1][Y0-0000000000000000-0-0] [lt=24] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=0, this=0x7effbcbf66e0, extra_info="") [2025-03-19 15:02:41.113321] INFO [LIB] stat (utility.h:1210) [96513][T1_ApplySrv2][T1][Y0-0000000000000000-0-0] [lt=20] [APPLY STAT CB EXECUTE TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=-565, this=0x7effbcbf6910, extra_info="") [2025-03-19 15:02:41.113341] INFO [LIB] stat (utility.h:1210) [96513][T1_ApplySrv2][T1][Y0-0000000000000000-0-0] [lt=20] [APPLY STAT CB TOTAL TIME](cur_stat_count=24, stat_interval=5000000, avg (count/cost)=104, this=0x7effbcbf6b40, extra_info="") [2025-03-19 15:02:41.119481] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAD-00062F0B1BE93B49-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=9, this=0x7efcf40b96a0, extra_info="palf_id:1002") [2025-03-19 15:02:41.119511] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAD-00062F0B1BE93B49-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=578, this=0x7efcf40b98d0, extra_info="palf_id:1002") [2025-03-19 15:02:41.120305] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAD-00062F0B1BE93B4A-0-0] [lt=10] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=8, this=0x7efdb10d36a0, extra_info="palf_id:1003") [2025-03-19 15:02:41.120328] INFO [LIB] stat (utility.h:1210) [2924119][T1004_L0_G1][T1004][YB420A686AAD-00062F0B1BE93B4A-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=572, this=0x7efdb10d38d0, extra_info="palf_id:1003") [2025-03-19 15:02:41.156976] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [802166][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=55] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1003, server_version={version:{val:1742367760958665000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=198303, in_cluster_service=true, cluster_version={val:1742367760757994000, v:0}, min_cluster_version={val:1742367760757994000, v:0}, max_cluster_version={val:1742367760757994000, v:0}, get_cluster_version_err=0, cluster_version_delta=398974, cluster_service_master="10.104.106.173:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=12613812, succ_cluster_heartbeat_count=10, cluster_heartbeat_interval=50000, local_cluster_version={val:1742367759157186000, v:0}, local_cluster_delta=1999782, force_self_check=false, weak_read_refresh_interval=100000) [2025-03-19 15:02:41.159169] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [802166][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=60] [PALF STAT GROUP LOG INFO](palf_id=1, self="10.104.106.173:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=11810, avg_group_log_size=621) [2025-03-19 15:02:41.159206] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [802166][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=33] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"10.104.106.173:2882", has_set_deleted:false}, append size=11810) [2025-03-19 15:02:41.159426] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [802139][T1003_LogIOCb0][T1003][Y0-0000000000000000-0-0] [lt=36] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="10.104.106.173:2882", committed size=11810) [2025-03-19 15:02:41.159764] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [2924099][T1003_L0_G1][T1003][YB420A686AAF-00062F088448A2D6-0-0] [lt=21] [PALF STAT LOG TASK TIME](palf_id=1, self="10.104.106.173:2882", total_slide_log_cnt=19, avg_log_gen_to_freeze_time=4, avg_log_gen_to_submit_time=57, avg_log_submit_to_flush_time=231, avg_log_submit_to_slide_time=567) [2025-03-19 15:02:41.160352] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [802132][T1003_TimerWK3][T1003][Y0-0000000000000000-0-0] [lt=40] sql audit evict task end(request_manager_->get_tenant_id()=1003, evict_high_mem_level=103079214, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=95668224) [2025-03-19 15:02:41.160403] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [802129][T1003_TimerWK0][T1003][Y0-0000000000000000-0-0] [lt=31] Construct Queue Num(construct_num=0, get_push_idx()=1659, get_cur_idx()=1651, get_pop_idx()=1651) [2025-03-19 15:02:41.177404] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [801889][T1004_TimerWK0][T1004][Y0-0000000000000000-0-0] [lt=33] sql audit evict task end(request_manager_->get_tenant_id()=1004, evict_high_mem_level=927712935, evict_high_size_level=884736, evict_batch_count=0, elapse_time=0, size_used=458752, mem_used=900529152) [2025-03-19 15:02:41.177458] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [801892][T1004_TimerWK3][T1004][Y0-0000000000000000-0-0] [lt=36] Construct Queue Num(construct_num=0, get_push_idx()=974, get_cur_idx()=966, get_pop_idx()=959) [2025-03-19 15:02:41.183928] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [784637][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=0) [2025-03-19 15:02:41.187167] INFO [LIB] stat (utility.h:1210) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=20] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=46, this=0x7efe9d9a7470, extra_info="") [2025-03-19 15:02:41.187203] INFO [LIB] stat (utility.h:1210) [801956][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x7efdd2dcc050, extra_info="id:1") [2025-03-19 15:02:41.187379] INFO try_report (ob_tc_stat.cpp:35) [784913][][T0][Y0-0000000000000000-0-0] [lt=28] QSched: |cfg: root:1,x,0 net_in:1,1249M,0 net_out:1,1249M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:6,x,0 t1m0gd:1,x,0 t1m1:6,x,0 t1m1gd:1,x,0 t1m2:6,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 ops:x obw:x t1m2g0:100,x,x t1m0g10000:100,x,0 t1m1g10000:100,x,0 t1m2g10000:100,x,0 t1002m0:6,x,0 t1002m0gd:1,x,0 t1002m1:6,x,0 t1002m1gd:1,x,0 t1002m2:6,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:6,x,0 t1001m0gd:1,x,0 t1001m1:6,x,0 t1001m1gd:1,x,0 t1001m2:6,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x t1004m0:12,x,0 t1004m0gd:1,x,0 t1004m1:12,x,0 t1004m1gd:1,x,0 t1004m2:12,x,x t1004m2gd:1,x,0 t1004m0g0:100,x,0 t1004m1g0:100,x,0 ops:x obw:x ops:x obw:x t1004m2g0:100,x,x t1003m0:12,x,0 t1003m0gd:1,x,0 t1003m1:12,x,0 t1003m1gd:1,x,0 t1003m2:12,x,x t1003m2gd:1,x,0 t1003m0g0:100,x,0 ips:x ibw:x t1003m1g0:100,x,0 ops:x obw:x t1003m2g0:100,x,x|stat: root:320K/74:14714 disk:320K/74:14714 t1m2:44K/10:13624 t1m2g0:44K/10:13624 t1004m2:196K/47:15430 t1004m2g0:196K/47:15430 t1003m2:80K/17:13373 t1003m2g0:80K/17:13373 |q0: 0 0 0 |q1: 0 0 0 [2025-03-19 15:02:41.187388] INFO [LIB] stat (utility.h:1210) [801900][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=198, this=0x7efe9d9b1c30, extra_info="palf_id:1") [2025-03-19 15:02:41.187424] INFO [LIB] stat (utility.h:1227) [801900][T1004_IOWorker][T1004][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=20, this=0x7efd20dfe530, extra_info="")