[2025-01-08 00:50:46.286349] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=105][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.296099] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [46193][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] clock out of order(cur_ts=1736326246296097, cur_ts_=1736326246275168, delta=20929) [2025-01-08 00:50:46.313611] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:46.321291] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.321326] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.331876] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58301][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD83-0-0] [lt=13] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326246427884, keepalive_ts:-1}) [2025-01-08 00:50:46.331928] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58301][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD83-0-0] [lt=49] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326246427884, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:46.331990] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=96, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:46.335276] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8289-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=19, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:46.335312] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8289-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4771, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:46.335486] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=2335, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.342577] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=121] [PALF STAT WRITE LOG COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=2070, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:46.342628] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=50] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10700) [2025-01-08 00:50:46.342652] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=157, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:46.342677] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=286] [PALF STAT FLUSH CB COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=35, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:46.344141] INFO [LIB] stat (utility.h:1210) [58321][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B07-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=7, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:46.344177] INFO [LIB] stat (utility.h:1210) [58321][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B07-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=3441, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:46.344190] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58321][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B07-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=19, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=109, avg_log_submit_to_flush_time=2340, avg_log_submit_to_slide_time=3331) [2025-01-08 00:50:46.344371] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10700, hit_cnt=19, read_cnt=19, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.344467] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=37] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326245690224003, v:0}, new_max_version={val:1736326245690224003, v:0}, last_min_version={val:1736326245477976003, v:0}, last_max_version={val:1736326245477976003, v:0}, total_time_us=5169, delta_us=654241) [2025-01-08 00:50:46.344496] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326245690224003, v:0}, min_version={val:1736326245690224003, v:0}, max_version={val:1736326245690224003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:46.345645] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC366D-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:46.345723] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC366D-0-0] [lt=68] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:46.351692] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=44] gc stale ls task succ [2025-01-08 00:50:46.354018] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:50:46.365934] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43AB-0-0] [lt=16] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:46.382958] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=20, avg_log_batch_cnt=1, total_group_log_size=11289, avg_group_log_size=564) [2025-01-08 00:50:46.382997] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=11289) [2025-01-08 00:50:46.386559] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.388420] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=109] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:46.388451] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-01-08 00:50:46.388461] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:46.388475] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:46.388489] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193750302122, flush_io_finished_ret_:0, flush_io_finished_round_:1905, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:46.388950] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=83] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:46.388973] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-01-08 00:50:46.388981] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:46.388990] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:46.389001] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193750302649, flush_io_finished_ret_:0, flush_io_finished_round_:3733, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:46.400087] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=45] ts response task statistics(alloc_count=59310, free_count=29654) [2025-01-08 00:50:46.405866] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1830, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:46.406069] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=199] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=5289) [2025-01-08 00:50:46.406095] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=275, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:46.406131] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:46.407447] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A5F-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:46.407471] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A5F-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3546, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:46.407482] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A5F-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=12, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=99, avg_log_submit_to_flush_time=2312, avg_log_submit_to_slide_time=3447) [2025-01-08 00:50:46.407589] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=5289, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.435568] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC828C-0-0] [lt=46] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=117, avg_log_submit_to_flush_time=3098, avg_log_submit_to_slide_time=4543) [2025-01-08 00:50:46.442966] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [50416][T1002_TimerWK1][T1001][YB42C0A89F89-00062B2D1D185376-0-0] [lt=37] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1736326246437667, can_tenant_elr:true}) [2025-01-08 00:50:46.443977] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:46.444692] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=38] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:46.458863] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=656101376, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=417640448, wash_ratio=6, tenant_id=1001) [2025-01-08 00:50:46.458984] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=75] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:50:46.459005] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2785, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:46.459077] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=15] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1112293376, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2108932096, wash_ratio=6, tenant_id=1002) [2025-01-08 00:50:46.459133] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=19] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:50:46.459147] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46098][TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=13] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:46.459292] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.459331] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.460675] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=211] PNIO [ratelimit] time: 1736326246460671, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007606, add_bytes: 0 [2025-01-08 00:50:46.467652] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1157, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:46.467708] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=53] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:46.467735] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=86, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:46.467805] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=63, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:46.486712] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.514667] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=84] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:46.519108] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.532446] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=77, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:46.532494] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:46.532516] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:46.532531] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:46.534059] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1742, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:46.534106] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=175, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:46.534147] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:46.536235] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC828F-0-0] [lt=18] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=9, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:46.536274] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC828F-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4306, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:46.541307] INFO pn_ratelimit (group.c:653) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=48] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:50:46.542020] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=53] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.542078] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58135][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB183-0-0] [lt=14] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1002, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}, {id:1001}]}) [2025-01-08 00:50:46.542105] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58135][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB183-0-0] [lt=20] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:46.542133] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58135][T1002_L0_G19][T1002][YB42C0A89F8B-00062B2D1EBBB183-0-0] [lt=25] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}) [2025-01-08 00:50:46.544299] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.550502] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1568, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:46.550545] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:46.550517] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=66551808) [2025-01-08 00:50:46.550576] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=86, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:46.550652] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=50, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:46.550823] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get_min_unreplayed_log_info(lsn={lsn:2170041}, scn={val:1736326245982947005, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326246469179, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2170041}, next_to_submit_scn_:{val:1736326245982947005, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2169919}, end_lsn:{lsn:2170041}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2170041}, readable_log_tail:{lsn:2170041}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64934727, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326245982947004, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326245982947004, v:0}, data_checksum:3348837161, flag:4734}}, init_mode_version:0, accumulate_checksum:2930602984, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326245982947004, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17740, miss_cnt:0, cache_read_size:2170041, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:46.551465] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:50:46.551472] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58329][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:46.551795] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:50:46.553816] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCB-0-0] [lt=27] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:50:46.553845] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCB-0-0] [lt=23] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=32) [2025-01-08 00:50:46.555265] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1038048, replace_num=15728) [2025-01-08 00:50:46.555308] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:50:46.566087] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43AF-0-0] [lt=29] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:46.567118] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=44] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:50:46.578085] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=124] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:46.586855] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.605003] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=77, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:46.637771] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:46.644007] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=144] get_max_applied_scn(scn={val:1736326246399779001, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18634142}, palf_committed_end_scn_:{val:1736326246399779002, v:0}, last_check_scn_:{val:1736326246399779002, v:0}, max_applied_cb_scn_:{val:1736326246399779001, v:0}}) [2025-01-08 00:50:46.644048] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18634142}, readable_log_tail:{lsn:18634142}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48470626, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:46.644290] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=216] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:50:46.666291] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43B1-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:46.666415] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43B1-0-0] [lt=47][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17829, log_task={header:{begin_lsn:{lsn:2170041}, end_lsn:{lsn:2170163}, log_id:17829, min_scn:{val:1736326246484620000, v:0}, max_scn:{val:1736326246484620000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2169919}, prev_proposal_id:1, committed_end_lsn:{lsn:2170041}, data_checksum:1715074492, accum_checksum:3448166050, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326246566051, freeze_ts:1736326246566051, submit_ts:1736326246566061, flushed_ts:1736326246569419, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:3358}, fs_cb_begin_ts=1736326246666345, log_life_time=100294) [2025-01-08 00:50:46.669520] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=114, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:46.669549] INFO [LIB] stat (utility.h:1210) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:46.686536] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB29F-0-0] [lt=37] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246678461, keepalive_ts:-1}) [2025-01-08 00:50:46.686586] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB29F-0-0] [lt=48] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246678461, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:46.686992] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=110][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.687106] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A0-0-0] [lt=17] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246678550, keepalive_ts:-1}) [2025-01-08 00:50:46.687123] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A0-0-0] [lt=16] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246678550, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:46.697018] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50502][T1002_L0_G0][T1002][YB42C0A89F8A-00062B2D1BC9C774-0-0] [lt=4] get number(ret=0, service_type_=0, range=1, base_id=1736326246696225000, start_id=1736326246696225000, end_id=1736326246696225001) [2025-01-08 00:50:46.713137] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=22] ====== tenant freeze timer task ====== [2025-01-08 00:50:46.715260] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:46.743922] INFO [STORAGE.TRANS] get_cluster_version (ob_tenant_weak_read_cluster_version_mgr.cpp:166) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=85] [WEAK_READ_SERVER_VERSION_MGR] compute version(tenant_id=1002, min_version={val:1736326245982947004, v:0}, base_version={val:1736326245982947002, v:0}, server_count=3, svr_array=[{addr:"192.168.159.137:2882", version:{val:1736326245982947004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326246743903, is_skipped:false}, {addr:"192.168.159.138:2882", version:{val:1736326245982947004, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326246766935, is_skipped:false}, {addr:"192.168.159.139:2882", version:{val:1736326246193063002, v:0}, valid_part_count:4, total_part_count:4, generate_tstamp:1736326246686030, is_skipped:false}], skip_server_count=0, skip_servers=[]) [2025-01-08 00:50:46.755474] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:50:46.765569] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=40] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:46.773826] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=47] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:46.785137] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10183) [2025-01-08 00:50:46.787139] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [50576][T1001_RFLWorker][T1001][YB42C0A89F89-00062B2D1C78458E-0-0] [lt=109] ObRemoteFetchWorker is running(thread_index=0) [2025-01-08 00:50:46.787432] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.787675] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0ED-0-0] [lt=12] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246779340, keepalive_ts:-1}) [2025-01-08 00:50:46.787719] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0ED-0-0] [lt=42] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326246779340, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:46.796619] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49CFF6-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:46.804322] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:46.808003] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=29] LogLoopThread round_cost_time(us)(round_cost_time=12) [2025-01-08 00:50:46.818469] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C7E-0-0] [lt=54][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:46.819718] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C7E-0-0] [lt=38][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:46.820544] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1736326246820541, bytes: 865262320, bw: 0.207148 MB/s, add_ts: 1009626, add_bytes: 219301 [2025-01-08 00:50:46.827650] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=43] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:46.827697] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=37] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=190) [2025-01-08 00:50:46.827736] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=110] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:46.827806] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=62] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:46.827821] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=13] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=184) [2025-01-08 00:50:46.834541] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2066) [2025-01-08 00:50:46.838382] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=224] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.844273] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4508) [2025-01-08 00:50:46.850671] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1002}, {id:1}, {id:1001}]) [2025-01-08 00:50:46.853266] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8534, offset=2420316}, end_cursor:ObLogCursor{file_id=1, log_id=8535, offset=2420621}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894648050, len:420, log_data_len:305, seq:8534, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:46.853621] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E38-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:46.853729] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E38-0-0] [lt=90] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326246696225001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326246696225001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:46.857623] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28B-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:46.857707] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28B-0-0] [lt=75] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326246484620002, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326246484620002, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:46.860588] INFO [LIB] stat (utility.h:1210) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E38-0-0] [lt=22] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=98, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:46.870719] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=33] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8535, offset=2420621}, end_cursor:ObLogCursor{file_id=1, log_id=8536, offset=2420918}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89464c050, len:4211, log_data_len:297, seq:8535, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:46.871452] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E38-0-0] [lt=21] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:46.871494] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E38-0-0] [lt=24] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326246696225001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326246696225001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:46.877971] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C28B-0-0] [lt=77] get number(ret=0, service_type_=0, range=1, base_id=1736326246874156000, start_id=1736326246874156001, end_id=1736326246874156002) [2025-01-08 00:50:46.883752] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58327][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B11-0-0] [lt=60] init point thread id with(&point=0x5572d50e1f00, idx_=983, point=[thread id=58327, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58327) [2025-01-08 00:50:46.884412] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=36] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8536, offset=2420918}, end_cursor:ObLogCursor{file_id=1, log_id=8537, offset=2421215}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894664050, len:3914, log_data_len:297, seq:8536, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:46.885872] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=71] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8537, offset=2421215}, end_cursor:ObLogCursor{file_id=1, log_id=8538, offset=2421512}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894650050, len:3617, log_data_len:297, seq:8537, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:46.886258] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17743, miss_cnt:0, cache_read_size:2170407, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:46.887541] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:46.892433] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-01-08 00:50:46.892461] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=104) [2025-01-08 00:50:46.897315] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.897583] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.897634] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.897711] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:46.900837] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:46.904891] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EBE-0-0] [lt=10] MediumCheckTask(cost_ts=9) [2025-01-08 00:50:46.916409] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=95] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6737457152, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:46.917665] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=45] ====== check clog disk timer task ====== [2025-01-08 00:50:46.917693] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:46.933447] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=2066, avg_group_log_size=206) [2025-01-08 00:50:46.933492] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2066) [2025-01-08 00:50:46.933513] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:46.939319] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2596, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:46.939381] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=52] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=230, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:46.939418] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=55] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:50:46.939437] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:46.939483] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:46.939579] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:46.944148] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=48] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:46.946886] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:46.947025] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:46.947984] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3679-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:46.950708] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58242][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=29] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:50:46.953642] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58245][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8EE4-0-0] [lt=48] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=13, avg_log_submit_to_flush_time=1765, avg_log_submit_to_slide_time=12466) [2025-01-08 00:50:46.953788] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=178] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.953829] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17657, miss_cnt:0, cache_read_size:2160082, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:46.955684] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:50:46.966819] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43B7-0-0] [lt=22] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:46.966849] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43B7-0-0] [lt=29] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=22647, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:46.966860] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43B7-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=21, avg_log_submit_to_flush_time=2022, avg_log_submit_to_slide_time=22626) [2025-01-08 00:50:46.966989] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:46.976256] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326236961449, end_time=1736326246976254, dag_status={}) [2025-01-08 00:50:46.976287] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=27] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976300] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976308] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976316] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976323] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976330] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976405] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=33] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976419] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=54] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976426] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:46.976433] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976445] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976453] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976461] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976470] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976478] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976485] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976493] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976501] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976509] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976517] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976525] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976533] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976541] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976549] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976557] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976564] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976572] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976580] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976588] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976596] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976604] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976611] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976619] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976627] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976635] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976642] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976650] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976658] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976666] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976674] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976682] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976689] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976697] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976705] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976712] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976720] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976728] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976787] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=58] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976796] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976804] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976812] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976820] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976828] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976836] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976844] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976852] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976859] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976867] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976875] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976882] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976890] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976898] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976906] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976913] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976921] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976929] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976936] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976944] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976952] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976960] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976967] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976975] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976983] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976991] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.976998] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977006] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977014] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977022] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977030] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977037] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977045] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977053] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:46.977062] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:50:46.977070] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:50:46.977077] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:50:46.977083] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:50:46.977088] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:50:46.977093] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:50:46.977098] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:50:46.977103] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:50:46.977108] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:50:46.977113] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:50:46.977119] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [46298][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:50:46.980466] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D2C-0-0] [lt=15] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:50:46.983750] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AB55-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.742487001419067383e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=387, Throttle Trigger(MB)=232, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.811310442503267950e+02) [2025-01-08 00:50:46.987526] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:46.987634] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44888, [delay/us]:prepare:8, schedule:63, submit:56, rt:1057, total:1185) [2025-01-08 00:50:46.987654] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:46.987728] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=58] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44893, [delay/us]: prepare:8, schedule:63, submit:56, device:1057, total:1185) [2025-01-08 00:50:46.987744] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:46.987783] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4742, iops:19, bw:89787, [delay/us]:prepare:7, schedule:137, submit:41, rt:1255, total:1441) [2025-01-08 00:50:46.987794] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:46.987831] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4743, avg_iops:19, avg_bw:89793, [delay/us]: prepare:7, schedule:137, submit:41, device:1255, total:1441) [2025-01-08 00:50:46.987842] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:46.987864] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20407, [delay/us]:prepare:7, schedule:91, submit:1, rt:1491, total:1591) [2025-01-08 00:50:46.987896] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:40, bw:163262, [delay/us]:prepare:6, schedule:126, submit:56, rt:1317, total:1507) [2025-01-08 00:50:46.987908] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:46.987937] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=25] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20408, [delay/us]: prepare:7, schedule:91, submit:1, device:1491, total:1591) [2025-01-08 00:50:46.987953] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4096, avg_iops:40, avg_bw:163268, [delay/us]: prepare:6, schedule:126, submit:56, device:1317, total:1507) [2025-01-08 00:50:46.987991] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=33] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:46.988011] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4081, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:267, total:274) [2025-01-08 00:50:46.988056] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:46.988090] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4081, [delay/us]: prepare:0, schedule:0, submit:600000000, device:267, total:274) [2025-01-08 00:50:46.988109] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:46.988125] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:46.988133] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:46.988141] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:46.988148] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=282kB/s, net_out=237kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:46.988622] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.005588] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=4386, avg_group_log_size=438) [2025-01-08 00:50:47.005644] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=51] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4386) [2025-01-08 00:50:47.005662] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:47.007723] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B87-0-0] [lt=82] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:47.012183] INFO [CLOG] run1 (ob_garbage_collector.cpp:1466) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] Garbage Collector is running(seq_=192, gc_interval=10000000) [2025-01-08 00:50:47.012228] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=25] self is leader, skip it(ls->get_ls_id()={id:1}) [2025-01-08 00:50:47.012243] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1601) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] self is leader, skip it(ls->get_ls_id()={id:1001}) [2025-01-08 00:50:47.015510] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] GC check ls in member list, skip it(id={id:1002}, leader="192.168.159.138:2882") [2025-01-08 00:50:47.016617] INFO [CLOG] handle_rpc_response_ (ob_garbage_collector.cpp:274) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=24] GC check ls in member list, skip it(id={id:1003}, leader="192.168.159.139:2882") [2025-01-08 00:50:47.016644] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1559) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=22] gc_check_member_list_ cost time(ret=0, time_us=4436) [2025-01-08 00:50:47.016660] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=0) [2025-01-08 00:50:47.018091] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:47.019466] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:47.020801] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:47.022087] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1755) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}) [2025-01-08 00:50:47.022111] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:47.022127] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=15] ls status is normal, skip(id={id:1001}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:47.022138] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1002}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:47.022149] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1777) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] ls status is normal, skip(id={id:1003}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1001}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1002}, ls_status_:1, gc_reason_:0}, {ls_id_:{id:1003}, ls_status_:1, gc_reason_:0}]) [2025-01-08 00:50:47.022160] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1833) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=11] execute_gc cost time(ret=0, time_us=50) [2025-01-08 00:50:47.022169] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyHandler start process [2025-01-08 00:50:47.022177] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=6] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-01-08 00:50:47.022258] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [50441][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=70] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-01-08 00:50:47.045176] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x5572d527f900, last_check_time:1736326247045172, last_idle_time:14974162262, last_thread_count:4}, *pool={name:TimerWK, this:0x5572d527f900, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:14974162262, tenant_id:500}, qsize=-4, current_idle_time=14974162262) [2025-01-08 00:50:47.045283] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=105] [thread pool stat](pool_stat={pool:0x2ad8885abcf0, last_check_time:1736326247045282, last_idle_time:5176894544, last_thread_count:2}, *pool={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5176894544, tenant_id:500}, qsize=-2, current_idle_time=5176894544) [2025-01-08 00:50:47.045297] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] [thread pool stat](pool_stat={pool:0x2ad8885acaf0, last_check_time:1736326247045297, last_idle_time:3743228072, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885acaf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3743228072, tenant_id:500}, qsize=-1, current_idle_time=3743228072) [2025-01-08 00:50:47.045308] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8881fc3b0, last_check_time:1736326247045307, last_idle_time:14904819990, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8881fc3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:14904819990, tenant_id:1}, qsize=-4, current_idle_time=14904819990) [2025-01-08 00:50:47.045318] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8885ad1f0, last_check_time:1736326247045317, last_idle_time:3655319780, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x2ad8885ad1f0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:3655319780, tenant_id:500}, qsize=-1, current_idle_time=3655319780) [2025-01-08 00:50:47.045327] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] [thread pool stat](pool_stat={pool:0x2ad8c0eb0e70, last_check_time:1736326247045327, last_idle_time:7695491913, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8c0eb0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7695491913, tenant_id:1002}, qsize=-4, current_idle_time=7695491913) [2025-01-08 00:50:47.045337] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:610) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] [thread pool stat](pool_stat={pool:0x2ad8f90b0e70, last_check_time:1736326247045337, last_idle_time:7664844199, last_thread_count:4}, *pool={name:TimerWK, this:0x2ad8f90b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:7664844199, tenant_id:1001}, qsize=-4, current_idle_time=7664844199) [2025-01-08 00:50:47.045348] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5176894544, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:47.045361] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:47.045373] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad90857db90) [2025-01-08 00:50:47.045381] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5176894544, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:47.050762] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:47.051987] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-01-08 00:50:47.067228] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=51] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:47.067267] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=26] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=92) [2025-01-08 00:50:47.067983] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2174747, this={dio_aligned_buf:{buf_write_offset:3989, buf_padding_size:107, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:26062, truncate_used_ts:5364}, log_block_size:67108864, io_fd:{first_id:-1, second_id:190, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2170880, buf_len=122, total_write_size=11806, total_write_size_after_dio=405504, ob_pwrite_used_ts=613786, count=97, each_pwrite_cost=6327) [2025-01-08 00:50:47.075946] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46508][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18577D-0-0] [lt=41][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193751989779, cache_obj->added_lc()=false, cache_obj->get_object_id()=13544, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:47.076681] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46502][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B18577E-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:47.078144] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=87][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:47.078277] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:47.078701] WDIAG get_refactored (ob_hashtable.h:1120) [46384][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185780-0-0] [lt=43][errcode=-4201] hashtable not init [2025-01-08 00:50:47.079255] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=24][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:47.088815] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.098845] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84146977}, readable_log_tail:{lsn:84146977}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:50062559, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:47.098942] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=95] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:50:47.099125] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=111] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:50:47.116998] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:47.118594] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=48] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=314570, clean_num=31457) [2025-01-08 00:50:47.129054] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=58][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:47.138169] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:47.139541] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=31] ====== checkpoint timer task ====== [2025-01-08 00:50:47.140506] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.140607] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=101] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.140720] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=26] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.140808] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=84] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:50:47.140828] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:47.140842] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326243478874001, v:0}) [2025-01-08 00:50:47.140850] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:50:47.140864] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:47.140883] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:47.140896] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:47.140915] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:47.140965] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=49] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:47.141822] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.141863] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=41] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.141907] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326241890747001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.141974] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=63] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:50:47.141988] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:50:47.142003] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:50:47.142018] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:50:47.142034] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2137625}, scn={val:1736326246900070001, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:47.143132] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:316K/74:143741 disk:316K/74:143741 t1m2:44K/10:190471 t1m2g0:44K/10:190471 t1002m2:188K/46:136122 t1002m2g0:188K/46:136122 t1001m2:84K/18:137250 t1001m2g0:84K/18:137250 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:47.149268] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=61] [PALF STAT APPEND COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:47.149706] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB36-0-0] [lt=21][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193752063540, cache_obj->added_lc()=false, cache_obj->get_object_id()=13545, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:47.152800] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50899][T1002_L0_G2][T1002][YB42C0A89F8B-00062B2D1EABD762-0-0] [lt=30] get_min_unreplayed_log_info(lsn={lsn:2160204}, scn={val:1736326246693924004, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326247051172, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2160204}, next_to_submit_scn_:{val:1736326246693924004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2160082}, end_lsn:{lsn:2160204}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2160204}, readable_log_tail:{lsn:2160204}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64944564, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326246693924003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326246693924003, v:0}, data_checksum:2124447694, flag:19657}}, init_mode_version:0, accumulate_checksum:4202015717, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326246693924003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17658, miss_cnt:0, cache_read_size:2160204, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:47.154942] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Slow write(duration=12079, write_len=4096) [2025-01-08 00:50:47.154966] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=23] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8538, offset=2421512}, end_cursor:ObLogCursor{file_id=1, log_id=8539, offset=2421809}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89465c050, len:3320, log_data_len:297, seq:8538, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.155069] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] ====== check clog disk timer task ====== [2025-01-08 00:50:47.155094] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=279) [2025-01-08 00:50:47.155306] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:50:47.155389] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326246900070001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:47.155415] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:47.155444] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=28] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:50:47.156014] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=48] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:50:47.156196] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.156302] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=104] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.156398] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326241475765004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.156472] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=70] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:50:47.156487] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:50:47.156505] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:50:47.156520] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:50:47.156535] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2170773}, scn={val:1736326246984891000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:47.157317] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=1864) [2025-01-08 00:50:47.159125] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=17] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8539, offset=2421809}, end_cursor:ObLogCursor{file_id=1, log_id=8540, offset=2422106}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894658050, len:3023, log_data_len:297, seq:8539, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.159277] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326246984891000, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:47.159382] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=98][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:47.159406] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=23] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:50:47.160281] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=59] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.160317] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=36] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:47.160360] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326241679125002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.160439] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=77] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:50:47.160454] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:50:47.160468] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:50:47.160482] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:50:47.160495] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2160204}, scn={val:1736326246693924003, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:47.162360] INFO [PALF] pread (log_reader.cpp:130) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=2048) [2025-01-08 00:50:47.163690] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=47] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8540, offset=2422106}, end_cursor:ObLogCursor{file_id=1, log_id=8541, offset=2422403}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894654050, len:2726, log_data_len:297, seq:8540, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.163843] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326246693924003, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:47.163874] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:47.163894] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:50:47.163904] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:50:47.170419] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC35F-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.180569] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=50] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:47.181884] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=28] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:47.182031] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=42] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326244258401000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:47.188916] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.195918] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE7B5-0-0] [lt=279] get number(ret=0, service_type_=0, range=1, base_id=1736326247195038000, start_id=1736326247195038000, end_id=1736326247195038001) [2025-01-08 00:50:47.201131] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=8] ====== traversal_flush timer task ====== [2025-01-08 00:50:47.201301] INFO [MDS] flush (mds_table_mgr.cpp:195) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854A1-0-0] [lt=41] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326246900070002, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.201500] INFO [MDS] flush (mds_table_mgr.cpp:172) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854A1-0-0] [lt=163] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246900070001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.201644] INFO [MDS] flush (mds_table_mgr.cpp:172) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854A1-0-0] [lt=114] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246984891000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.201851] INFO [MDS] flush (mds_table_mgr.cpp:172) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854A1-0-0] [lt=76] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246693924003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.201930] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50417][T1002_TimerWK2][T1002][YB42C0A89F89-00062B2D1C6854A1-0-0] [lt=76] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:50:47.214367] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=39] gc stale ls task succ [2025-01-08 00:50:47.219738] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:47.219765] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:47.223346] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB86FFB-0-0] [lt=31][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.223391] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:47.223403] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:47.234481] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=95] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:47.235834] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:47.235947] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=41] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326245675194000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:47.235975] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=25] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326245675194000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1, freeze_info:{latest_snapshot_gc_scn:{val:1736326245675194000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:50:47.243815] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=25] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:47.243838] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:50:47.245136] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848DF-0-0] [lt=8] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:47.245163] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848DF-0-0] [lt=24] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:47.248520] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC367F-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:47.248561] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC367F-0-0] [lt=39] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=13064, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:47.249023] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:47.250029] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58330][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:47.250540] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:50:47.256531] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:47.257379] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=17] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:50:47.270817] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82A6-0-0] [lt=24] init point thread id with(&point=0x5572d50e1f80, idx_=985, point=[thread id=58329, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58329) [2025-01-08 00:50:47.272351] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C95-0-0] [lt=17] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:47.272398] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C95-0-0] [lt=43] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:47.272447] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=88] gc stale ls task succ [2025-01-08 00:50:47.272733] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E1-0-0] [lt=22] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:47.272787] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E1-0-0] [lt=37] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326247195038000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326247195038000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:47.283456] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5E1-0-0] [lt=157] get number(ret=0, service_type_=0, range=1, base_id=1736326247282506000, start_id=1736326247282506000, end_id=1736326247282506001) [2025-01-08 00:50:47.285160] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540A-0-0] [lt=15][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.289055] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.293763] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:42973173}], scn_array=[cnt:1, {val:1736326247282506000, v:0}], curr_size=750, accum_size=9596, time_cost=10012) [2025-01-08 00:50:47.293950] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=24] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:47.300767] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58321][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF24-0-0] [lt=67] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326247291830, keepalive_ts:-1}) [2025-01-08 00:50:47.300799] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58321][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF24-0-0] [lt=31] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326247291830, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:47.319224] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=70] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:47.319246] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.319284] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=63] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.327645] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=167] current monitor number(seq_=-1) [2025-01-08 00:50:47.333137] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:47.334433] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=47] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:47.334542] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=38] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326245564663000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:47.335535] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] tx gc loop thread is running(MTL_ID()=1002) [2025-01-08 00:50:47.335695] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=156] try gc retain ctx [2025-01-08 00:50:47.335707] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] try to retry start_working [2025-01-08 00:50:47.335739] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326246900070003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326237108711001, v:0}, keep_alive_scn_:{val:1736326241890747004, v:0}, update_ts_:1736326247334299}) [2025-01-08 00:50:47.336561] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:47.336718] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=67, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:47.336764] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] finish update min start scn(max_decided_scn={val:1736326246900070003, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326237108711000, v:0}, keep_alive_scn_:{val:1736326241890747004, v:0}, update_ts_:1736326247334299}) [2025-01-08 00:50:47.337489] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246900070001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:47.337633] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=59] finish update min start scn(max_decided_scn={val:1736326246984891001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326239968575000, v:0}, keep_alive_scn_:{val:1736326244981816000, v:0}, update_ts_:1736326247334299}) [2025-01-08 00:50:47.338557] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246984891000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:47.338626] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=59] finish update min start scn(max_decided_scn={val:1736326247195038000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326238175372002, v:0}, keep_alive_scn_:{val:1736326243184127002, v:0}, update_ts_:1736326247334299}) [2025-01-08 00:50:47.339333] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=11] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246693924003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326244396051002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:47.342339] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8EEF-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:47.342375] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8EEF-0-0] [lt=34] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4028, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:47.342893] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=81] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1634, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:47.349339] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3681-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:47.349373] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3681-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:47.351345] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=48] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=2324, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:47.351371] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10185) [2025-01-08 00:50:47.351389] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=379, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:47.351773] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=35, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:47.351831] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=28] gc stale ls task succ [2025-01-08 00:50:47.353051] INFO [LIB] stat (utility.h:1210) [58321][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37B1-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:47.353074] INFO [LIB] stat (utility.h:1210) [58321][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37B1-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4175, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:47.353084] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58321][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37B1-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=163, avg_log_gen_to_submit_time=270, avg_log_submit_to_flush_time=2912, avg_log_submit_to_slide_time=3905) [2025-01-08 00:50:47.353291] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=46] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326246693924003, v:0}, new_max_version={val:1736326246693924003, v:0}, last_min_version={val:1736326246484620003, v:0}, last_max_version={val:1736326246484620003, v:0}, total_time_us=5152, delta_us=659364) [2025-01-08 00:50:47.353321] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326246693924003, v:0}, min_version={val:1736326246693924003, v:0}, max_version={val:1736326246693924003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:47.353394] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10185, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:47.356186] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:50:47.367532] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43BF-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:47.374347] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50583][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=48] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:47.388672] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:47.388725] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=45] temporary file module use no blocks [2025-01-08 00:50:47.388738] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:47.388752] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:47.388815] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=61] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193751302361, flush_io_finished_ret_:0, flush_io_finished_round_:1906, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:47.389241] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.401660] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=86] get number(ret=0, service_type_=0, range=1, base_id=1736326247401274000, start_id=1736326247401274000, end_id=1736326247401274001) [2025-01-08 00:50:47.407483] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1519, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:47.407578] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=93] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3963) [2025-01-08 00:50:47.407603] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=407, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:47.407632] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=175] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=53, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:47.409855] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58330][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6701-0-0] [lt=26] init point thread id with(&point=0x5572d50e1fc0, idx_=986, point=[thread id=58330, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58330) [2025-01-08 00:50:47.409895] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A69-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:47.409919] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A69-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3391, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:47.409929] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A69-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=5, avg_log_gen_to_submit_time=79, avg_log_submit_to_flush_time=2076, avg_log_submit_to_slide_time=3311) [2025-01-08 00:50:47.410004] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3963, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:47.411886] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=99] ====== traversal_flush timer task ====== [2025-01-08 00:50:47.412113] INFO [MDS] flush (mds_table_mgr.cpp:195) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EBF-0-0] [lt=42] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324381352256005, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325284636804001, v:0}}, }, max_consequent_callbacked_scn={val:1736326247382286000, v:0}, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259284252000, latest_log_ts:{val:1736326242623339001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.412352] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50559][T1001_TimerWK1][T1001][YB42C0A89F89-00062B2D1DA84EBF-0-0] [lt=194] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:50:47.417957] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=28] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-01-08 00:50:47.417992] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] start lib cache evict(tenant_id=1001, mem_hold=46137344, mem_limit=53687090, cache_obj_num=130, cache_node_num=130) [2025-01-08 00:50:47.418006] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=46137344, mem_limit=53687090, cache_obj_num=130, cache_node_num=130) [2025-01-08 00:50:47.418017] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:47.420641] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=130, alloc_obj_list=[{obj_id:372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202612}, {obj_id:501, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190240}, {obj_id:519, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201328}, {obj_id:522, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:635, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194594}, {obj_id:641, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:707, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:742, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202872}, {obj_id:743, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:784, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189618}, {obj_id:860, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:1430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201780}, {obj_id:1439, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:1455, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187012}, {obj_id:1458, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:1460, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194163}, {obj_id:1462, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179084}, {obj_id:1465, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:1504, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179168}, {obj_id:1524, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193377}, {obj_id:1769, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:1785, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190237}, {obj_id:1822, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1829, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1865, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1885, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1966, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:1967, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200576}, {obj_id:1968, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184986}, {obj_id:1969, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178024}, {obj_id:1970, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1971, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199088}, {obj_id:1972, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198336}, {obj_id:1973, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216784}, {obj_id:1974, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203406}, {obj_id:1975, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1976, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:1977, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178824}, {obj_id:1978, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:1979, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:1980, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:1981, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192722}, {obj_id:1982, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179834}, {obj_id:1983, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204622}, {obj_id:1984, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199497}, {obj_id:1985, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:1986, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:1987, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:1988, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434150}, {obj_id:1990, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328075}, {obj_id:1991, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451878}, {obj_id:1992, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:1993, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:1994, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:1995, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:1996, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:1997, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:1998, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:1999, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:2000, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:2001, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:2002, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:2003, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:2004, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:2005, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:2006, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:2007, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:2008, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:2009, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:2010, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:2011, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:2012, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:2013, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:2016, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2017, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:2018, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2019, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:2020, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2021, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:2022, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2023, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2024, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2025, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2027, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:2028, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194023}, {obj_id:2029, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:2031, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233845}, {obj_id:2033, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:2036, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:2038, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192539}, {obj_id:2041, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203353}, {obj_id:2042, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:2043, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184136}, {obj_id:2044, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:2045, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:2046, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:2047, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184568}, {obj_id:2048, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2049, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203354}, {obj_id:2050, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2051, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:2052, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:2053, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182356}, {obj_id:2054, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:2055, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2056, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2057, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:2058, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:2059, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:2060, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:2061, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185345}, {obj_id:2062, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:2063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:2064, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:2065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:2066, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:2067, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:2068, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180377}, {obj_id:2069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184081}, {obj_id:2070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:2071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:2072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191558}, {obj_id:2073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:2074, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:221591}, {obj_id:2075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:2076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185446}, {obj_id:2077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:2078, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:2079, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:2080, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}]) [2025-01-08 00:50:47.421036] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=390] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:47.429490] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9] ====== tenant freeze timer task ====== [2025-01-08 00:50:47.429522] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70285300, tenant_id=1001) [2025-01-08 00:50:47.429595] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=63] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:50:47.443956] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58265][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8EF1-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=11, avg_log_gen_to_submit_time=93, avg_log_submit_to_flush_time=2633, avg_log_submit_to_slide_time=4397) [2025-01-08 00:50:47.447937] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=91] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:47.448747] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:47.459557] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.459567] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=41] ====== tenant freeze timer task ====== [2025-01-08 00:50:47.459623] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.466021] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46493][T1_L0_G0][T1][YB42C0A89F8A-00062B2D16C9BEC0-0-0] [lt=53] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:50:47.468527] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1736326247468523, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007852, add_bytes: 0 [2025-01-08 00:50:47.471287] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2167, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:47.471339] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:47.471364] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=369, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:47.471436] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=41, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:47.489389] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.519308] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.519739] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:47.521683] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=101] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=19, avg_log_batch_cnt=1, total_group_log_size=10307, avg_group_log_size=542) [2025-01-08 00:50:47.521717] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10774) [2025-01-08 00:50:47.536258] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=204] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=52, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:47.536829] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=547] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:50:47.536852] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:50:47.536868] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:47.540095] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8EF4-0-0] [lt=48] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:50:47.542382] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=32] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.542394] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=75] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2655, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:47.542446] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=537, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:47.542592] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:47.542616] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5464, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:47.544394] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.553434] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:47.553605] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:47.553847] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=43] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:47.556346] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:50:47.556612] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58265][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=30] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:50:47.567820] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43C3-0-0] [lt=69] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:47.578446] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:47.589519] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=90][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.600658] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B89-0-0] [lt=18] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:47.607659] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=55, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:47.638310] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=63] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:47.638624] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2142200, this={dio_aligned_buf:{buf_write_offset:114, buf_padding_size:3982, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:22333, truncate_used_ts:5026}, log_block_size:67108864, io_fd:{first_id:-1, second_id:187, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=8192, aligned_block_offset=2138112, buf_len=122, total_write_size=10342, total_write_size_after_dio=360448, ob_pwrite_used_ts=761632, count=85, each_pwrite_cost=8960) [2025-01-08 00:50:47.649741] INFO [CLOG] is_in_sync_ (palf_handle_impl.cpp:5315) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=26] is_in_sync(ret=0, palf_id=1003, is_in_sync=true, is_use_cache=true, remote_check_is_log_sync=false, leader_max_scn={val:18446744073709551615, v:3}, local_max_scn={val:1736326247195038004, v:0}) [2025-01-08 00:50:47.651588] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58332][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:47.651678] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25] worker thread created(tenant_->id()=1, group_id_=1, token=3) [2025-01-08 00:50:47.651729] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1733, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:47.651757] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:50:47.651776] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=117, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:47.651827] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=54, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:47.668065] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43C5-0-0] [lt=46] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:47.670679] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=19] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=91, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:47.670707] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=27] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=5, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:47.671252] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=8] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:47.671304] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=36] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=91) [2025-01-08 00:50:47.678754] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=96] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):4608, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):279, sequence:1}) [2025-01-08 00:50:47.683671] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=115] get_max_applied_scn(scn={val:1736326247585417000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:42976401}, palf_committed_end_scn_:{val:1736326247585417000, v:0}, last_check_scn_:{val:1736326247585417000, v:0}, max_applied_cb_scn_:{val:1736326247585417000, v:0}}) [2025-01-08 00:50:47.683724] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:42976401}, readable_log_tail:{lsn:42976401}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:24128367, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:47.689607] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.720358] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=86] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:47.756837] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:50:47.765769] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=41] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:47.774395] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=84] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:47.788984] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58321][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37B8-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10187) [2025-01-08 00:50:47.789754] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.796274] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49CFF9-0-0] [lt=26][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.804754] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=91] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:47.822734] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50377][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=117] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}, {1001,0,False,{False,},False,False,True}, {1002,1,False,{False,},False,False,True}, {1003,1,False,{False,},False,False,True}]) [2025-01-08 00:50:47.828571] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=36] PNIO [ratelimit] time: 1736326247828567, bytes: 865585587, bw: 0.305837 MB/s, add_ts: 1008026, add_bytes: 323267 [2025-01-08 00:50:47.829880] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C7F-0-0] [lt=56][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.832856] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C7F-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:47.834390] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A6D-0-0] [lt=15] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12287, add_timestamp:1736326247834373}) [2025-01-08 00:50:47.835346] INFO [SHARE.LOCATION] fetch_vtable_location_ (ob_vtable_location_service.cpp:281) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594D-0-0] [lt=23] fetch virtual table location with tenant(rpc tenant=0, tenant_id=1, table_id=12287, vtable_type={tenant_id:1, type:4}) [2025-01-08 00:50:47.840376] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594D-0-0] [lt=26] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12287, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:47.840421] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594D-0-0] [lt=42] success to process renew task(task={tenant_id:1, table_id:12287, add_timestamp:1736326247834373}, locations=[{table_id:12287, partition_id:0, partition_cnt:3, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:1, partition_cnt:3, replica_locations:[{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}, {table_id:12287, partition_id:2, partition_cnt:3, replica_locations:[{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326247840347, sql_renew_time:0, is_mark_fail:false}]) [2025-01-08 00:50:47.840519] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:50:47.850445] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=13] ====== traversal_flush timer task ====== [2025-01-08 00:50:47.850849] INFO [MDS] flush (mds_table_mgr.cpp:195) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F8578F-0-0] [lt=32] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:2, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736323479158254002, v:0}}, }, max_consequent_callbacked_scn={val:1736326247644191000, v:0}, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259254996000, latest_log_ts:{val:1736326242681587001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:47.850998] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F8578F-0-0] [lt=142] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-01-08 00:50:47.851037] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}, {id:1003}, {id:1001}, {id:1002}]) [2025-01-08 00:50:47.851369] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:47.852805] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3539) [2025-01-08 00:50:47.858396] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1736326247844395000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0, epoch_tstamp:1736326247858361}, version_delta=13999) [2025-01-08 00:50:47.859967] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=81] ls blacklist refresh finish(cost_time=27029) [2025-01-08 00:50:47.889945] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.890502] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=104] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8541, offset=2422403}, end_cursor:ObLogCursor{file_id=1, log_id=8542, offset=2422708}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894660050, len:2429, log_data_len:305, seq:8541, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.891024] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E39-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:47.891067] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E39-0-0] [lt=25] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326247401274004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326247401274004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:47.896575] INFO [LIB] stat (utility.h:1210) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E39-0-0] [lt=54] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=100, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:47.896958] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=33] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.897261] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=54] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.897435] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.897459] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:47.898378] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28C-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:47.898419] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28C-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326247485899003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326247485899003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:47.902417] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=51] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326247902396], gts:1736326247901652000, latest_srr:[mts=1736326247902396]}) [2025-01-08 00:50:47.902449] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:47.902472] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326247902461], gts:1736326247901652001, latest_srr:[mts=1736326247902461]}) [2025-01-08 00:50:47.902714] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326247902702], gts:1736326247901652000, latest_srr:[mts=1736326247902702]}) [2025-01-08 00:50:47.906530] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=38] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8542, offset=2422708}, end_cursor:ObLogCursor{file_id=1, log_id=8543, offset=2423005}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894668050, len:2124, log_data_len:297, seq:8542, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.906892] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E39-0-0] [lt=82] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:47.906958] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E39-0-0] [lt=48] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326247401274004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326247401274004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:47.921060] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=116] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:47.923537] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8543, offset=2423005}, end_cursor:ObLogCursor{file_id=1, log_id=8544, offset=2423302}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89466c050, len:1827, log_data_len:297, seq:8543, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.929825] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=50] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8544, offset=2423302}, end_cursor:ObLogCursor{file_id=1, log_id=8545, offset=2423599}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894674050, len:1530, log_data_len:297, seq:8544, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:47.930306] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17754, miss_cnt:0, cache_read_size:2171749, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:47.936714] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1211, avg_group_log_size=121) [2025-01-08 00:50:47.936747] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1211) [2025-01-08 00:50:47.936764] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:47.939516] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:47.939986] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82B9-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1333) [2025-01-08 00:50:47.941237] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1211) [2025-01-08 00:50:47.941341] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=49, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:47.947872] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=63] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:47.948275] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=64] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:47.949143] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=12478464) [2025-01-08 00:50:47.950065] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:47.951444] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC368D-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:47.953308] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58321][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=48] worker thread exit(tenant_->id()=1001, workers_.get_size()=7) [2025-01-08 00:50:47.957067] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=46] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:50:47.976607] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82BB-0-0] [lt=40] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=50, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:47.976657] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82BB-0-0] [lt=49] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=12922, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:47.976625] INFO [PALF] read (log_cache.cpp:103) [50480][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:47.976670] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82BB-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=40, avg_log_submit_to_flush_time=1852, avg_log_submit_to_slide_time=12882) [2025-01-08 00:50:47.984264] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB38-0-0] [lt=89][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193752898096, cache_obj->added_lc()=false, cache_obj->get_object_id()=13546, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:47.987143] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46477][T1_L0_G5][T1][YB42C0A89F8A-00062B2D1829F451-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1736326247986924000, start_id=1736326247986924000, end_id=1736326247986924001) [2025-01-08 00:50:47.990290] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:47.990342] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44934, [delay/us]:prepare:26, schedule:572, submit:179, rt:1315, total:2093) [2025-01-08 00:50:47.990357] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:47.990474] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44935, [delay/us]: prepare:26, schedule:572, submit:179, device:1315, total:2093) [2025-01-08 00:50:47.990491] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:47.990585] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4577, iops:17, bw:77606, [delay/us]:prepare:8, schedule:250, submit:236, rt:1573, total:2068) [2025-01-08 00:50:47.990600] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:47.990640] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4578, avg_iops:17, avg_bw:77606, [delay/us]: prepare:8, schedule:250, submit:236, device:1573, total:2068) [2025-01-08 00:50:47.990711] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=65] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:47.990738] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:7, bw:28589, [delay/us]:prepare:13, schedule:60, submit:1, rt:3562, total:3637) [2025-01-08 00:50:47.990761] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:171538, [delay/us]:prepare:7, schedule:180, submit:348, rt:1382, total:1919) [2025-01-08 00:50:47.990770] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_READ_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:3, bw:12252, [delay/us]:prepare:1, schedule:129, submit:215, rt:218, total:565) [2025-01-08 00:50:47.990779] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:47.990810] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:7, avg_bw:28589, [delay/us]: prepare:13, schedule:60, submit:1, device:3562, total:3637) [2025-01-08 00:50:47.990826] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL READ, avg_size:4096, avg_iops:3, avg_bw:12252, [delay/us]: prepare:1, schedule:129, submit:215, device:218, total:565) [2025-01-08 00:50:47.990833] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:171537, [delay/us]: prepare:7, schedule:180, submit:348, device:1382, total:1919) [2025-01-08 00:50:47.990885] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:47.990909] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4084, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:251, total:259) [2025-01-08 00:50:47.990935] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:47.991008] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=67] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4084, [delay/us]: prepare:0, schedule:0, submit:600000000, device:251, total:259) [2025-01-08 00:50:47.991028] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:47.991043] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:47.991052] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:47.991059] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:47.991067] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=257kB/s, net_out=249kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:47.991176] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:47.994044] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB3B-0-0] [lt=40] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.791824254989624023e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=895, Throttle Trigger(MB)=537, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.762973993267839390e+01) [2025-01-08 00:50:47.996046] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB3D-0-0] [lt=65] get number(ret=0, service_type_=0, range=1, base_id=1736326247995951000, start_id=1736326247995951000, end_id=1736326247995951001) [2025-01-08 00:50:47.999876] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA6707-0-0] [lt=27] init point thread id with(&point=0x5572d50e2040, idx_=988, point=[thread id=58332, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58332) [2025-01-08 00:50:48.000125] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=30] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:50:48.036308] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3B-0-0] [lt=14] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326248132022, keepalive_ts:-1}) [2025-01-08 00:50:48.036346] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3B-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326248132022, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:48.039287] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2026, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:48.039342] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=377, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:48.045410] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50588][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=40] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f95562b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:50:48.045995] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53] no need to update min/max version(new_version={val:1736326247644191000, v:0}, min_version={val:1736326247644191000, v:0}, max_version={val:1736326247644191000, v:0}) [2025-01-08 00:50:48.050892] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC368F-0-0] [lt=15] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=2, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=2193, avg_log_submit_to_slide_time=31815) [2025-01-08 00:50:48.050998] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:48.051094] INFO [PALF] next (palf_iterator.h:183) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=85] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17667, miss_cnt:0, cache_read_size:2161302, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:48.053670] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:48.054081] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:48.067737] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:400) [46100][TimerWK2][T0][YB42C0A89F89-00062B2D17E85432-0-0] [lt=69] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-01-08 00:50:48.078639] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=94] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:48.091297] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=70][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.101767] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326244089394, last_first_register_time_us=1736324449183670, register_time_us=1736326244089394, curr_time_us=1736326248101764) [2025-01-08 00:50:48.101898] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326248101805, last_submit_register_req_time_us=1736326248101805, register_parent_reason=1) [2025-01-08 00:50:48.101921] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:50:48.108451] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4104, avg_group_log_size=373) [2025-01-08 00:50:48.108481] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4104) [2025-01-08 00:50:48.108499] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:48.121429] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=278] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:48.138501] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:48.147828] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326248145826000, start_id=1736326248145826000, end_id=1736326248145826001) [2025-01-08 00:50:48.148292] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:324K/76:189932 disk:324K/76:189932 t1m2:48K/11:430733 t1m2g0:48K/11:430733 t1002m2:200K/48:117835 t1002m2g0:200K/48:117835 t1001m2:76K/17:237689 t1001m2g0:76K/17:237689 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:48.153745] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT APPEND COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:48.157402] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:50:48.176793] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB3E-0-0] [lt=37][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193753090625, cache_obj->added_lc()=false, cache_obj->get_object_id()=13547, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:48.184322] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=107] no need to update min/max version(new_version={val:1736326247747226000, v:0}, min_version={val:1736326247747226000, v:0}, max_version={val:1736326247747226000, v:0}) [2025-01-08 00:50:48.186656] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC362-0-0] [lt=29][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.191432] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.201935] INFO [STORAGE] scheduler_transfer_handler_ (ob_transfer_service.cpp:202) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F0-0-0] [lt=21] start do transfer handler(ls_id_array_=[{id:1}, {id:1001}, {id:1002}, {id:1003}]) [2025-01-08 00:50:48.205461] WDIAG [SHARE] get_by_ls_id_ (ob_transfer_task_operator.cpp:754) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=29][errcode=-4018] dest ls transfer task not found(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1002, ls_id={id:1}) [2025-01-08 00:50:48.205587] WDIAG [SHARE] get_by_src_ls (ob_transfer_task_operator.cpp:189) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=21][errcode=-4018] failed to get by ls id(ret=-4018, tenant_id=1002, src_ls={id:1}) [2025-01-08 00:50:48.205605] WDIAG [STORAGE] fetch_transfer_task_from_inner_table_by_src_ls_ (ob_transfer_handler.cpp:246) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=18][errcode=-4018] failed to get transfer task(ret=-4018, tenant_id=1002, src_ls_id={id:1}) [2025-01-08 00:50:48.206821] WDIAG [SHARE] get_by_ls_id_ (ob_transfer_task_operator.cpp:754) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=8][errcode=-4018] dest ls transfer task not found(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1002, ls_id={id:1}) [2025-01-08 00:50:48.206896] WDIAG [SHARE] get_by_dest_ls (ob_transfer_task_operator.cpp:207) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=18][errcode=-4018] failed to get by ls id(ret=-4018, tenant_id=1002, dest_ls={id:1}) [2025-01-08 00:50:48.206913] WDIAG [STORAGE] fetch_transfer_task_from_inner_table_by_dest_ls_ (ob_transfer_handler.cpp:272) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=16][errcode=-4018] failed to get transfer task by dest ls(ret=-4018, tenant_id=1002, dest_ls_id={id:1}) [2025-01-08 00:50:48.209018] INFO [STORAGE] get_need_backfill_tx_tablets_ (ob_transfer_backfill_tx.cpp:239) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=8] statistics the time that get needed to backfill tablets(ls id={id:1}, cost_time=2091) [2025-01-08 00:50:48.209039] INFO [STORAGE] do_worker_transfer_ (ob_transfer_handler.cpp:2386) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=16] do worker transfer(ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326245477976002, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.210451] INFO [STORAGE] do_transfer_handler_ (ob_transfer_service.cpp:237) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F2-0-0] [lt=69] [TRANSFER] transfer handler process(ret=0, ls_id={id:1}) [2025-01-08 00:50:48.213594] WDIAG [SHARE] get_by_ls_id_ (ob_transfer_task_operator.cpp:754) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=17][errcode=-4018] dest ls transfer task not found(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1002, ls_id={id:1001}) [2025-01-08 00:50:48.213680] WDIAG [SHARE] get_by_src_ls (ob_transfer_task_operator.cpp:189) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=27][errcode=-4018] failed to get by ls id(ret=-4018, tenant_id=1002, src_ls={id:1001}) [2025-01-08 00:50:48.213696] WDIAG [STORAGE] fetch_transfer_task_from_inner_table_by_src_ls_ (ob_transfer_handler.cpp:246) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=15][errcode=-4018] failed to get transfer task(ret=-4018, tenant_id=1002, src_ls_id={id:1001}) [2025-01-08 00:50:48.214499] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=24] gc stale ls task succ [2025-01-08 00:50:48.214765] WDIAG [SHARE] get_by_ls_id_ (ob_transfer_task_operator.cpp:754) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=8][errcode=-4018] dest ls transfer task not found(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tenant_id=1002, ls_id={id:1001}) [2025-01-08 00:50:48.214840] WDIAG [SHARE] get_by_dest_ls (ob_transfer_task_operator.cpp:207) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=18][errcode=-4018] failed to get by ls id(ret=-4018, tenant_id=1002, dest_ls={id:1001}) [2025-01-08 00:50:48.214911] WDIAG [STORAGE] fetch_transfer_task_from_inner_table_by_dest_ls_ (ob_transfer_handler.cpp:272) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=70][errcode=-4018] failed to get transfer task by dest ls(ret=-4018, tenant_id=1002, dest_ls_id={id:1001}) [2025-01-08 00:50:48.214984] INFO [STORAGE] get_need_backfill_tx_tablets_ (ob_transfer_backfill_tx.cpp:239) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=13] statistics the time that get needed to backfill tablets(ls id={id:1001}, cost_time=57) [2025-01-08 00:50:48.214995] INFO [STORAGE] do_worker_transfer_ (ob_transfer_handler.cpp:2386) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=11] do worker transfer(ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246900070001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326245477976002, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.216347] INFO [STORAGE] do_transfer_handler_ (ob_transfer_service.cpp:237) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F4-0-0] [lt=56] [TRANSFER] transfer handler process(ret=0, ls_id={id:1001}) [2025-01-08 00:50:48.217653] INFO [STORAGE] get_need_backfill_tx_tablets_ (ob_transfer_backfill_tx.cpp:239) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F6-0-0] [lt=17] statistics the time that get needed to backfill tablets(ls id={id:1002}, cost_time=55) [2025-01-08 00:50:48.217673] INFO [STORAGE] do_worker_transfer_ (ob_transfer_handler.cpp:2386) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F6-0-0] [lt=18] do worker transfer(ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246984891000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326245477976002, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.218938] INFO [STORAGE] do_transfer_handler_ (ob_transfer_service.cpp:237) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F6-0-0] [lt=56] [TRANSFER] transfer handler process(ret=0, ls_id={id:1002}) [2025-01-08 00:50:48.219920] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:48.219945] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:48.220255] INFO [STORAGE] get_need_backfill_tx_tablets_ (ob_transfer_backfill_tx.cpp:239) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F8-0-0] [lt=17] statistics the time that get needed to backfill tablets(ls id={id:1003}, cost_time=115) [2025-01-08 00:50:48.220273] INFO [STORAGE] do_worker_transfer_ (ob_transfer_handler.cpp:2386) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F8-0-0] [lt=17] do worker transfer(ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246693924003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326245477976002, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.221536] INFO [STORAGE] do_transfer_handler_ (ob_transfer_service.cpp:237) [50469][T1002_TransferS][T1002][YB42C0A89F89-00062B2D21B845F8-0-0] [lt=56] [TRANSFER] transfer handler process(ret=0, ls_id={id:1003}) [2025-01-08 00:50:48.224998] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87001-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.225066] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:48.225080] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:48.255073] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=42] set tenant work area memory(tenant_id=1001, wa_pctg=5, limit=157286400, ret=0) [2025-01-08 00:50:48.255315] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=30] set tenant work area memory(tenant_id=1002, wa_pctg=5, limit=161061270, ret=0) [2025-01-08 00:50:48.256771] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=51] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:48.264819] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [46200][Blacklist][T0][Y0-0000000000000000-0-0] [lt=26] blacklist_loop exec finished(cost_time=21, is_enabled=true, send_cnt=0) [2025-01-08 00:50:48.274750] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C96-0-0] [lt=14] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:48.274790] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C96-0-0] [lt=37] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:48.274828] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:50:48.274920] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:48.274943] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:50:48.274951] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:48.274965] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:48.274979] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193752188332, flush_io_finished_ret_:0, flush_io_finished_round_:1911, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:48.287281] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540B-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.291599] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:48.291637] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.297327] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:50:48.297441] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [46312][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:50:48.301995] INFO [CLOG] update_upstream (ob_remote_location_adaptor.cpp:91) [50440][T1002_LogRessvr][T1002][YB42C0A89F89-00062B2D1BA8458E-0-0] [lt=23] primary tenant, just skip [2025-01-08 00:50:48.303346] EDIAG [SHARE.CONFIG] check_limit (ob_server_config.cpp:446) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] the hold memory of tenant_500 is over the reserved memory(hold=629149696, reserved=0) BACKTRACE:0x810107f 0x7b89ba6 0x7ee92d4 0x7ee8ac6 0x7ee8768 0x7ee8bfb 0x19343c4f 0x193439f6 0x186d6a1d 0x7b75587 0x7b27c01 0x1f9588e5 0x1f9566ee 0x2ad878b3fea5 0x2ad878e5296d [2025-01-08 00:50:48.303377] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:32) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=29] === Run print tenant memory usage task === [2025-01-08 00:50:48.303424] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:102) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=13] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 145,612,800 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9,223,372,036,854,775,807 mem_tenant_hold= 631,246,848 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 15,572,992 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1,736,326,248,302,215 active_memstore_used= 142,606,336 total_memstore_used= 142,606,336 total_memstore_hold= 142,606,336 memstore_freeze_trigger_limit= 160,801,160 memstore_limit= 858,993,440 mem_tenant_limit= 2,147,483,648 mem_tenant_hold= 1,038,594,048 max_mem_memstore_can_get_now= 804,005,886 memstore_alloc_pos= 218,103,808 memstore_frozen_pos= 75,497,472 memstore_reclaimed_pos= 75,497,472 [TENANT_MEMORY] tenant_id= 1,001 now= 1,736,326,248,302,215 active_memstore_used= 102,760,448 total_memstore_used= 102,760,448 total_memstore_hold= 102,760,448 memstore_freeze_trigger_limit= 70,285,300 memstore_limit= 429,496,720 mem_tenant_limit= 1,073,741,824 mem_tenant_hold= 656,101,376 max_mem_memstore_can_get_now= 351,426,548 memstore_alloc_pos= 102,760,448 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1,002 now= 1,736,326,248,302,215 active_memstore_used= 123,731,968 total_memstore_used= 123,731,968 total_memstore_hold= 123,731,968 memstore_freeze_trigger_limit= 241,092,180 memstore_limit= 1,288,490,160 mem_tenant_limit= 3,221,225,472 mem_tenant_hold= 1,110,196,224 max_mem_memstore_can_get_now= 1,205,460,968 memstore_alloc_pos= 123,731,968 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-01-08 00:50:48.303722] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:114) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=10] [CHUNK_MGR] limit= 15,032,385,536 hold= 3,658,604,544 total_hold= 3,856,662,528 used= 3,467,284,480 freelists_hold= 191,320,064 total_maps= 1,624 total_unmaps= 232 large_maps= 120 large_unmaps= 0 huge_maps= 11 huge_unmaps= 5 resident_size= 0 unmanaged_memory_size= 145,612,800 virtual_memory_used= 4,829,216,768 [CHUNK_MGR] 2 MB_CACHE: hold= 88,080,384 count= 42 pushes= 1,803,412 pops= 1,803,370 maps= 1,493 unmaps= 227 [CHUNK_MGR] 4 MB_CACHE: hold= 9,035,776 count= 3 pushes= 6,152 pops= 6,149 maps= 53 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 26,030,080 count= 6 pushes= 506 pops= 500 maps= 23 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 25,247,744 count= 4 pushes= 3,058 pops= 3,054 maps= 6 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 10,305,536 count= 1 pushes= 8 pops= 7 maps= 19 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 1 pops= 1 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 15,822,848 count= 1 pushes= 4 pops= 3 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 16,797,696 count= 1 pushes= 6 pops= 5 maps= 10 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 0 unmaps= 0 [2025-01-08 00:50:48.303750] INFO print (ob_malloc_time_monitor.cpp:37) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=10] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 6144, delta_count= 2, avg_cost_time= 3072 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 375968, delta_count= 1, avg_cost_time= 375968 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-01-08 00:50:48.305952] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E2-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:48.306017] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E2-0-0] [lt=47] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326247697287004, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326247697287004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:48.319313] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.319357] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.322039] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=108] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:48.326067] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=20] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:48.337288] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB55-0-0] [lt=46] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:48.337345] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8A-00062B2D1869BB55-0-0] [lt=49] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:48.337868] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=75, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:48.352017] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=15] gc stale ls task succ [2025-01-08 00:50:48.352975] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3695-0-0] [lt=45] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:48.353026] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3695-0-0] [lt=42] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:48.353057] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3695-0-0] [lt=16] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=13, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:48.353067] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3695-0-0] [lt=10] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=29855, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:48.353114] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3695-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17759, log_task={header:{begin_lsn:{lsn:2161546}, end_lsn:{lsn:2161668}, log_id:17759, min_scn:{val:1736326248201055000, v:0}, max_scn:{val:1736326248201055000, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2161424}, prev_proposal_id:1, committed_end_lsn:{lsn:2161546}, data_checksum:940009244, accum_checksum:4186338349, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326248250882, freeze_ts:1736326248250882, submit_ts:1736326248250892, flushed_ts:1736326248254901, gen_to_freeze cost time:0, gen_to_submit cost time:10, submit_to_flush cost time:4009}, fs_cb_begin_ts=1736326248353047, log_life_time=102165) [2025-01-08 00:50:48.353895] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=361] [PALF STAT WRITE LOG COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=2151, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:48.354120] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=222] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9051) [2025-01-08 00:50:48.354152] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=563, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:48.354365] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=17] [PALF STAT FLUSH CB COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=61, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:48.357630] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=109] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:50:48.357658] INFO [LIB] stat (utility.h:1210) [58207][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37C1-0-0] [lt=14] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=35, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:48.357691] INFO [LIB] stat (utility.h:1210) [58207][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37C1-0-0] [lt=32] [PALF STAT LOG LIFE TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=4412, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:48.357702] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58207][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37C1-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=16, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=126, avg_log_submit_to_flush_time=2842, avg_log_submit_to_slide_time=4285) [2025-01-08 00:50:48.357775] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=25] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9051, hit_cnt=16, read_cnt=16, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:48.358165] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326247697287003, v:0}, new_max_version={val:1736326247697287003, v:0}, last_min_version={val:1736326247401274004, v:0}, last_max_version={val:1736326247401274004, v:0}, total_time_us=7286, delta_us=660876) [2025-01-08 00:50:48.358309] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=139] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326247697287003, v:0}, min_version={val:1736326247697287003, v:0}, max_version={val:1736326247697287003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:48.368617] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43D3-0-0] [lt=180] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:48.389076] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=93] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:48.389115] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=32] temporary file module use no blocks [2025-01-08 00:50:48.389126] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:48.389139] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:48.389152] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193752302755, flush_io_finished_ret_:0, flush_io_finished_round_:1907, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:48.389545] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:48.389584] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] temporary file module use no blocks [2025-01-08 00:50:48.389592] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:48.389602] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:48.389613] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193752303070, flush_io_finished_ret_:0, flush_io_finished_round_:3735, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:48.391703] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.437573] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23] worker thread began to shrink(tenant_->id()=1001, group_id_=20, token=0) [2025-01-08 00:50:48.442025] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F10-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=14, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:48.442062] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F10-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=5706, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:48.442416] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:48.443306] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58034][T1001_L0_G20][T1001][Y0-0000000000000000-0-0] [lt=28] worker thread exit(tenant_->id()=1001, workers_.get_size()=1) [2025-01-08 00:50:48.446549] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=56] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:48.450367] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=131] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:48.450536] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=61] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:48.452372] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=23] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324343432440, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad9083f5a70, log_storage_:{log_tail:{lsn:42984152}, readable_log_tail:{lsn:42984152}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1001/1/log", dir_fd:173, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:24120616, block_header_serialize_buf_:0x2ad9083f6738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:48.452484] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=110] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:50:48.452518] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=0, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=0, replayed_log_size(MB)=0, last_submitted_log_size(MB)=0, last_replayed_log_size(MB)=0, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:50:48.453466] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=70] clear dtl interm result cost(us)(clear_cost=14, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:50:48.456281] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3697-0-0] [lt=128][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17760, log_task={header:{begin_lsn:{lsn:2161668}, end_lsn:{lsn:2161790}, log_id:17760, min_scn:{val:1736326248201055001, v:0}, max_scn:{val:1736326248201055001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2161546}, prev_proposal_id:1, committed_end_lsn:{lsn:2161668}, data_checksum:940009244, accum_checksum:1993249381, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326248352968, freeze_ts:1736326248352968, submit_ts:1736326248353019, flushed_ts:1736326248355569, gen_to_freeze cost time:0, gen_to_submit cost time:51, submit_to_flush cost time:2550}, fs_cb_begin_ts=1736326248456269, log_life_time=103301) [2025-01-08 00:50:48.458200] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.459352] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58033][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEBD-0-0] [lt=31] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:48.459391] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58033][T1_L0_G19][T1][YB42C0A89F8B-00062B2D187BAEBD-0-0] [lt=31] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:48.459571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.468315] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:685) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=27] clear dtl interm result cost(us)(clear_cost=3678, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-01-08 00:50:48.469277] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43D5-0-0] [lt=131][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17847, log_task={header:{begin_lsn:{lsn:2172237}, end_lsn:{lsn:2172359}, log_id:17847, min_scn:{val:1736326247986924003, v:0}, max_scn:{val:1736326247986924003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2172115}, prev_proposal_id:1, committed_end_lsn:{lsn:2172237}, data_checksum:1715074492, accum_checksum:3601824095, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326248368601, freeze_ts:1736326248368601, submit_ts:1736326248368749, flushed_ts:1736326248370041, gen_to_freeze cost time:0, gen_to_submit cost time:148, submit_to_flush cost time:1292}, fs_cb_begin_ts=1736326248469264, log_life_time=100663) [2025-01-08 00:50:48.469354] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58333][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:48.469545] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=55] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:50:48.475903] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=90] PNIO [ratelimit] time: 1736326248475900, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007377, add_bytes: 0 [2025-01-08 00:50:48.491784] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.514157] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2240, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:48.514245] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=87] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4104) [2025-01-08 00:50:48.514310] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=164, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:48.514711] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:48.514752] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4214, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:48.514766] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=7, avg_log_gen_to_submit_time=76, avg_log_submit_to_flush_time=2603, avg_log_submit_to_slide_time=4138) [2025-01-08 00:50:48.514800] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=30, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:48.514862] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4104, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:48.519441] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=85] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.522502] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:48.526430] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=16, avg_log_batch_cnt=1, total_group_log_size=9051, avg_group_log_size=565) [2025-01-08 00:50:48.526458] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9051) [2025-01-08 00:50:48.539244] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:48.539296] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:48.539317] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:48.539333] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:48.541768] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:48.542529] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.543917] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F12-0-0] [lt=84] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=28, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:48.543944] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F12-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=5096, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:48.544404] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.544505] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F13-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=109, avg_log_submit_to_flush_time=2790, avg_log_submit_to_slide_time=5375) [2025-01-08 00:50:48.552618] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC3699-0-0] [lt=156] init point thread id with(&point=0x5572d50e2080, idx_=989, point=[thread id=58333, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58333) [2025-01-08 00:50:48.553921] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=36] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:48.554463] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=80] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:48.557877] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:50:48.559415] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCC-0-0] [lt=23] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:50:48.559447] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCC-0-0] [lt=24] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=34) [2025-01-08 00:50:48.570353] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2073, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:48.570417] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=62] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:50:48.570446] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=150, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:48.570520] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=190] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=59, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:48.572472] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58289][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D5E2-0-0] [lt=9] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:50:48.579022] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=46] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:48.580008] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:62, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:2012642216, recv_hp_rpc_cnt:5121, recv_np_rpc_cnt:102763, recv_lp_rpc_cnt:0, recv_mysql_cnt:9246, recv_task_cnt:3700, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:10715977, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=7 cnt[3]=0 cnt[4]=0 cnt[5]=115021 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:25, queue_size:0, recv_req_cnt:1823, min_worker_cnt:1, max_worker_cnt:1, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736322581404009}{group_id:15, queue_size:0, recv_req_cnt:14, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921930638}{group_id:1, queue_size:0, recv_req_cnt:93030, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736326247650243}{group_id:10, queue_size:0, recv_req_cnt:11085, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322521798718}{group_id:20, queue_size:0, recv_req_cnt:1242, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326240198662}{group_id:19, queue_size:0, recv_req_cnt:2057, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326228853642}{group_id:2, queue_size:0, recv_req_cnt:29786, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322516773859}{group_id:9, queue_size:0, recv_req_cnt:11811, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736324926078623}{group_id:27, queue_size:0, recv_req_cnt:4932, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322586395050}{group_id:5, queue_size:0, recv_req_cnt:26471, min_worker_cnt:3, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:3, nesting_worker_cnt:0, token_change:1736322519297548}, rpc_stat_info: pcode=0x51c:cnt=308 pcode=0x1502:cnt=210 pcode=0x717:cnt=189 pcode=0x710:cnt=67 pcode=0x150b:cnt=60, token_change_ts:1736322519613412, tenant_role:1}) [2025-01-08 00:50:48.580949] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=937] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504286865, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:13917211, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:8129, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3545766, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info: pcode=0x5c0:cnt=23, token_change_ts:1736322519621656, tenant_role:0}) [2025-01-08 00:50:48.581634] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=685] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:1, }}, mode:0, create_timestamp:1736322504277777, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:20, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:13089351, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3532597, workers:12, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1736322519623778, tenant_role:0}) [2025-01-08 00:50:48.582259] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=623] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:23, min_worker_cnt:6, max_worker_cnt:150, stopped:0, worker_us:863494491, recv_hp_rpc_cnt:3668, recv_np_rpc_cnt:37254, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:764502, workers:6, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=64531 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:1, queue_size:0, recv_req_cnt:66684, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:6, nesting_worker_cnt:0, token_change:1736326247950061}{group_id:20, queue_size:0, recv_req_cnt:850, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:0, token_change:1736326248437570}{group_id:19, queue_size:0, recv_req_cnt:2144, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326218958545}{group_id:2, queue_size:0, recv_req_cnt:15181, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324403889286}{group_id:5, queue_size:0, recv_req_cnt:18209, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736324926170680}, rpc_stat_info: pcode=0x1502:cnt=350 pcode=0x51c:cnt=247 pcode=0x717:cnt=183 pcode=0x710:cnt=98 pcode=0x150b:cnt=60, token_change_ts:1736324333815589, tenant_role:1}) [2025-01-08 00:50:48.582958] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2679) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=697] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"4.000000000000000000e+00", unit_max_cpu:"4.000000000000000000e+00", total_worker_cnt:60, min_worker_cnt:18, max_worker_cnt:150, stopped:0, worker_us:9225092389, recv_hp_rpc_cnt:77, recv_np_rpc_cnt:34281, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:2891189, workers:18, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=4 cnt[3]=0 cnt[4]=0 cnt[5]=24661 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:{group_id:28, queue_size:0, recv_req_cnt:1, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:0, nesting_worker_cnt:9, token_change:1736325389949252}{group_id:18, queue_size:0, recv_req_cnt:6887, min_worker_cnt:1, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326208268040}{group_id:15, queue_size:0, recv_req_cnt:2, min_worker_cnt:2, max_worker_cnt:2, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:2, nesting_worker_cnt:0, token_change:1736324921931101}{group_id:1, queue_size:0, recv_req_cnt:147538, min_worker_cnt:8, max_worker_cnt:8, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:8, nesting_worker_cnt:0, token_change:1736326248468954}{group_id:20, queue_size:0, recv_req_cnt:404, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326242277705}{group_id:19, queue_size:0, recv_req_cnt:396, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:1, nesting_worker_cnt:0, token_change:1736326204280596}{group_id:2, queue_size:0, recv_req_cnt:36376, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324397452974}{group_id:27, queue_size:0, recv_req_cnt:14, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736325331432599}{group_id:5, queue_size:0, recv_req_cnt:10880, min_worker_cnt:4, max_worker_cnt:150, multi_level_queue_:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , worker_cnt:4, nesting_worker_cnt:0, token_change:1736324400125993}, rpc_stat_info: pcode=0x1502:cnt=436 pcode=0x51c:cnt=400 pcode=0x1501:cnt=199 pcode=0x717:cnt=192 pcode=0x1516:cnt=186, token_change_ts:1736324331584682, tenant_role:1}) [2025-01-08 00:50:48.591963] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.597857] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=12] BlockGCTimerTask success(ret=0, cost_time_us=10, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.159.137:2882", log_dir:"/ob/data/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):104, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-01-08 00:50:48.604676] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:191) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=852][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-01-08 00:50:48.638712] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=69] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:48.639995] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=30] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:48.640828] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2493, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:48.640961] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=946, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:48.643614] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=36] [APPLY STAT CB APPEND COST TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=148, this=0x2ad8fc468280, extra_info="") [2025-01-08 00:50:48.643640] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=26] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=4666, this=0x2ad8fc4684b0, extra_info="") [2025-01-08 00:50:48.643649] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8fc4686e0, extra_info="") [2025-01-08 00:50:48.643657] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=-1226, this=0x2ad8fc468910, extra_info="") [2025-01-08 00:50:48.643664] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB TOTAL TIME](cur_stat_count=29, stat_interval=5000000, avg (count/cost)=3588, this=0x2ad8fc468b40, extra_info="") [2025-01-08 00:50:48.655699] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:50:48.665372] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58330][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=35] worker thread exit(tenant_->id()=1, workers_.get_size()=3) [2025-01-08 00:50:48.669657] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43D9-0-0] [lt=36] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:48.670431] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1736326238656814, end_time=1736326248670429, dag_status={}) [2025-01-08 00:50:48.670534] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=53] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670630] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=95] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670639] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670647] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670654] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670661] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670668] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670675] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670681] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:48.670689] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670701] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670709] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670717] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670725] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670733] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670741] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670749] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670757] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670765] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670773] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670781] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670789] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670797] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670845] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=47] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670856] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670864] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670871] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670880] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670888] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670896] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670904] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670912] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670920] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670928] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670936] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.670943] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671050] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=105] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671060] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671068] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671076] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671084] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671092] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671100] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671108] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671116] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671124] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671132] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671140] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671148] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671156] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671165] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671173] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671257] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=83] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671269] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671278] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671285] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671293] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671301] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671309] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671317] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671324] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671333] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671340] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671348] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671356] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671364] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671372] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671380] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671388] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671395] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671403] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671411] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671419] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671427] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671434] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671442] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671451] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671459] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671466] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671474] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671482] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671490] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:48.671500] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:50:48.671508] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:50:48.671516] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:50:48.671521] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:50:48.671526] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:50:48.671532] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:50:48.671537] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:50:48.671542] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:50:48.671547] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:50:48.671552] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:50:48.671558] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50410][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:50:48.681973] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46380][T1_L0_G0][T1][YB42C0A89F8B-00062B2D199BBDB7-0-0] [lt=42][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193753595804, cache_obj->added_lc()=false, cache_obj->get_object_id()=13548, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:48.685246] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=91] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1736326248386431000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=298803, in_cluster_service=true, cluster_version={val:1736326248350690000, v:0}, min_cluster_version={val:1736326248350690000, v:0}, max_cluster_version={val:1736326248350690000, v:0}, get_cluster_version_err=0, cluster_version_delta=334544, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18494, succ_cluster_heartbeat_count=21, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325654363217000, v:0}, local_cluster_delta=594322017, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:48.692097] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.698899] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=32] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-01-08 00:50:48.700827] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=39] get number(ret=0, service_type_=0, range=1, base_id=1736326248699757000, start_id=1736326248699757000, end_id=1736326248699757001) [2025-01-08 00:50:48.700991] INFO [LIB] stat (utility.h:1210) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=20] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=59, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:48.722870] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:48.726597] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] tx gc loop thread is running(MTL_ID()=1001) [2025-01-08 00:50:48.726645] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=45] try gc retain ctx [2025-01-08 00:50:48.726709] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=63] try to retry start_working [2025-01-08 00:50:48.726746] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=6] finish update min start scn(max_decided_scn={val:1736326248586728000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326238498963001, v:0}, keep_alive_scn_:{val:1736326243676959001, v:0}, update_ts_:1736326248726426}) [2025-01-08 00:50:48.727913] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=93] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259284252000, latest_log_ts:{val:1736326242623339001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.740414] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46232][][T0][Y0-0000000000000000-0-0] [lt=18] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:48.740469] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2747) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", [SlidingWindow]={palf_id:1, self:"192.168.159.137:2882", lsn_allocator:{max_log_id:44332, max_lsn:84153042, max_scn:{val:1736326248699757000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:84153042}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:42210002}}, last_submit_lsn:{lsn:84152497}, last_submit_end_lsn:{lsn:84153042}, last_submit_log_id:44332, last_submit_log_pid:1, max_flushed_lsn:{lsn:84152497}, max_flushed_end_lsn:{lsn:84153042}, max_flushed_log_pid:1, committed_end_lsn:{lsn:84153042}, last_slide_log_id:44332, last_slide_scn:{val:1736326248699757000, v:0}, last_slide_lsn:{lsn:84152497}, last_slide_end_lsn:{lsn:84153042}, last_slide_log_pid:1, last_slide_log_accum_checksum:195906776, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2ad8aecd4530}, [StateMgr]={this:0x2ad8aecd8830, self:"192.168.159.137:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"192.168.159.137:2882", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:44333, is_changing_config_with_arb:false, reconfirm_start_time_us:1736322518838592, palf_role_change_cb:0x2ad8aecdf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.159.137:2882", alive_paxos_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:1}, persistent_config_version:{proposal_id:1, config_seq:2}, ms_ack_list:0, resend_config_version:{proposal_id:1, config_seq:2}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"sys_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2ad8aecd5c70}, [ModeMgr]={palf_id:1, self:"192.168.159.137:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:1736322519038872, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:0, min_block_min_scn:{val:2, v:0}, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:0, config_seq:1}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:2}, log_sync_memberlist:3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_mode_pid:1}, log_snapshot_meta:{version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/meta", dir_fd:153, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2ad8aecdaa38, flashback_version:0}, log_storage:{log_tail:{lsn:84153042}, readable_log_tail:{lsn:84153042}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:50056494, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1736322518838046, this:0x2ad8aecd8f70}, [Reconfirm]={palf_id:1, self:"192.168.159.137:2882", state:"FINISHED", new_proposal_id:1, prepare_log_ack_list:2"192.168.159.139:2882";"192.168.159.138:2882";, curr_paxos_follower_list:2{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"192.168.159.139:2882", majority_max_accept_pid:0, majority_max_lsn:{lsn:0}, saved_end_lsn:{lsn:0}, last_submit_prepare_req_time_us:1736322518839181, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2ad8aecd89b0}) [2025-01-08 00:50:48.741025] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2751) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=550] [PALF_DUMP](palf_id=1, self="192.168.159.137:2882", ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248706534, last_flushed_end_lsn:{lsn:84153042}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248704977, last_flushed_end_lsn:{lsn:84153042}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248704866, last_flushed_end_lsn:{lsn:84153042}}]) [2025-01-08 00:50:48.741051] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4325) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23] current log_task status(palf_id=1, self="192.168.159.137:2882", log_id=44333, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.159.137:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248706534, last_flushed_end_lsn:{lsn:84153042}}, {member:{server:"192.168.159.138:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248704977, last_flushed_end_lsn:{lsn:84153042}}, {member:{server:"192.168.159.139:2882", timestamp:1, flag:0}, last_ack_time_us:1736326248704866, last_flushed_end_lsn:{lsn:84153042}}]) [2025-01-08 00:50:48.741421] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=26] ====== tenant freeze timer task ====== [2025-01-08 00:50:48.742459] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:42985452}], scn_array=[cnt:1, {val:1736326248586728001, v:0}], curr_size=113, accum_size=2002, time_cost=14146) [2025-01-08 00:50:48.743853] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [50525][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=93] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:50:48.745672] INFO [LIB] do_thread_recycle (threads.cpp:163) [46232][][T0][Y0-0000000000000000-0-0] [lt=82] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:48.751447] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:1736326248201055003, v:0}, total_part_count:4, valid_inner_part_count:1, valid_user_part_count:3}, server_version_delta=550382, in_cluster_service=true, cluster_version={val:1736326247901652003, v:0}, min_cluster_version={val:1736326247901652003, v:0}, max_cluster_version={val:1736326247901652003, v:0}, get_cluster_version_err=0, cluster_version_delta=849785, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=18058, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325772082473004, v:0}, local_cluster_delta=476668964, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:48.755923] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=2169, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:48.755966] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1333) [2025-01-08 00:50:48.755987] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=378, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:48.756040] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=64, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:48.759120] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=7] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1053776, replace_num=15728) [2025-01-08 00:50:48.759149] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:50:48.761805] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D1708513F-0-0] [lt=38] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:48.765958] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=45] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:48.769102] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=33] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=578, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:48.769129] INFO [LIB] stat (utility.h:1210) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:48.774607] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=38] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:48.791526] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58207][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B30-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=9042) [2025-01-08 00:50:48.794665] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.800960] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49CFFC-0-0] [lt=27][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.803880] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] LogLoopThread round_cost_time(us)(round_cost_time=27) [2025-01-08 00:50:48.804058] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=80][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:42986154}], scn_array=[cnt:1, {val:1736326248785299000, v:0}], curr_size=588, accum_size=3179, time_cost=16141) [2025-01-08 00:50:48.805009] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=28] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:48.812415] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:106) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] tx gc loop thread is running(MTL_ID()=1) [2025-01-08 00:50:48.812465] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:113) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=48] try gc retain ctx [2025-01-08 00:50:48.812476] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:120) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try to retry start_working [2025-01-08 00:50:48.812504] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] get_max_applied_scn(scn={val:1736326248747804000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:84153628}, palf_committed_end_scn_:{val:1736326248747804000, v:0}, last_check_scn_:{val:1736326248747804000, v:0}, max_applied_cb_scn_:{val:1736326248747804000, v:0}}) [2025-01-08 00:50:48.812528] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736322513821128, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8aecdba70, log_storage_:{log_tail:{lsn:84153628}, readable_log_tail:{lsn:84153628}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1736323654894295001, v:0}, curr_block_id:1, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1/1/log", dir_fd:154, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:50055908, block_header_serialize_buf_:0x2ad8aecdc738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:48.812636] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1053) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=104] finish update min start scn(max_decided_scn={val:1736326248747804000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1736326238679896001, v:0}, keep_alive_scn_:{val:1736326243686095001, v:0}, update_ts_:1736326248811270}) [2025-01-08 00:50:48.813704] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:271) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259254996000, latest_log_ts:{val:1736326242681587001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-01-08 00:50:48.830414] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1736326248830410, bytes: 865840509, bw: 0.242665 MB/s, add_ts: 1001843, add_bytes: 254922 [2025-01-08 00:50:48.841702] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C80-0-0] [lt=68][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.842503] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=268] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:48.843627] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [46318][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30] LogLoopThread round_cost_time(us)(round_cost_time=634) [2025-01-08 00:50:48.845993] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C80-0-0] [lt=39][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:48.848362] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5178613819, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:48.848396] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:48.848408] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad9085995f0) [2025-01-08 00:50:48.848417] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5178613819, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:48.851270] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}, {id:1003}]) [2025-01-08 00:50:48.892671] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=95] ====== check clog disk timer task ====== [2025-01-08 00:50:48.892788] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=65] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=104) [2025-01-08 00:50:48.894764] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=91][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.897274] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=75] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.897365] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.897411] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.897478] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:48.904040] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=75] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:48.904382] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38] gts statistics(tenant_id=1002, gts_rpc_cnt=10, get_gts_cache_cnt=502, get_gts_with_stc_cnt=6, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=4, try_wait_gts_elapse_cnt=0) [2025-01-08 00:50:48.911649] INFO [STORAGE.COMPACTION] schedule_all_tablets_medium (ob_tenant_tablet_scheduler.cpp:1597) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8574F-0-0] [lt=17] start schedule all tablet merge(merge_version=1) [2025-01-08 00:50:48.912132] INFO [STORAGE] get_adaptive_merge_reason (ob_partition_merge_policy.cpp:1719) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8574F-0-0] [lt=95] Check tablet adaptive merge reason(ret=0, ls_id={id:1}, tablet_id={id:60476}, reason=0, tablet_analyzer={tablet_stat:{ls_id:1, tablet_id:60476, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, total_tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_small_tenant:false, boost_factor:"1.000000000000000000e+00"}, crazy_medium_flag=false) [2025-01-08 00:50:48.913853] INFO [COORDINATOR] refresh (ob_leader_coordinator.cpp:146) [46269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=25] refresh __all_ls_election_reference_info success(ret=0, ret="OB_SUCCESS", *new_all_ls_election_reference_info=[{1,0,False,{False,},False,False,True}]) [2025-01-08 00:50:48.915644] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4519) [2025-01-08 00:50:48.923307] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=61] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:48.925940] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=35] ====== check clog disk timer task ====== [2025-01-08 00:50:48.925982] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=37] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:48.927607] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=166] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=16, stat_interval=1000000, avg (count/cost)=76, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:48.935228] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=42] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8545, offset=2423599}, end_cursor:ObLogCursor{file_id=1, log_id=8546, offset=2423904}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894670050, len:1233, log_data_len:305, seq:8545, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:48.936101] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:48.936297] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=42] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326248403491004, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326248403491004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:48.940595] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:48.940626] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:48.940642] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:48.941364] INFO [SHARE] update_tenant_recovery_status_in_trans (ob_tenant_info_proxy.cpp:491) [50387][T1002_TeRec][T1002][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=8] tenant scn gap info(IS_MAX_GAP_REACHED=false, REAL_GAP=0, MAX_GAP=900000000000, new_sync_scn={val:1736326247401274004, v:0}, new_replayable_scn={val:1736326247401274004, v:0}, new_readable_scn={val:1736326247401274004, v:0}, old_tenant_info={tenant_id:1002, tenant_role:"PRIMARY", switchover_status:"NORMAL", switchover_epoch:0, sync_scn:1736326246484620002, replayable_scn:1736326246484620002, readable_scn:1736326246484620002, recovery_until_scn:4611686018427387903, log_mode:"NOARCHIVELOG", max_ls_id:1003, restore_data_mode:{restore data mode:"NORMAL"}}) [2025-01-08 00:50:48.941803] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50387][T1002_TeRec][T1001][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=29] get number(ret=0, service_type_=0, range=1, base_id=1736326248940813000, start_id=1736326248940813000, end_id=1736326248940813001) [2025-01-08 00:50:48.942692] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28D-0-0] [lt=59] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:48.942730] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28D-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326248488325003, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326248488325003, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:48.943083] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=22] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:48.945100] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=186] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:48.945563] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:48.945833] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=223] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=51, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:48.946072] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82D8-0-0] [lt=21] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:50:48.947035] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:48.950244] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=73] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8546, offset=2423904}, end_cursor:ObLogCursor{file_id=1, log_id=8547, offset=2424201}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894678050, len:928, log_data_len:297, seq:8546, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:48.951255] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:48.951386] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=45] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:48.951607] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=23] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:48.952646] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58275][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=108] worker thread exit(tenant_->id()=1001, workers_.get_size()=6) [2025-01-08 00:50:48.955287] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36A1-0-0] [lt=47] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1211) [2025-01-08 00:50:48.956200] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=68] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:48.956250] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3A-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326248403491004, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326248403491004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:48.960060] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:50:48.964450] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18637240}], scn_array=[cnt:1, {val:1736326248903338000, v:0}], curr_size=122, accum_size=122, time_cost=21916) [2025-01-08 00:50:48.965237] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:48.970699] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8547, offset=2424201}, end_cursor:ObLogCursor{file_id=1, log_id=8548, offset=2424498}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89467c050, len:631, log_data_len:297, seq:8547, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:48.971947] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50623][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C28D-0-0] [lt=23] get number(ret=0, service_type_=0, range=1, base_id=1736326248971489000, start_id=1736326248971489000, end_id=1736326248971489001) [2025-01-08 00:50:48.972390] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17765, miss_cnt:0, cache_read_size:2173091, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:48.974818] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8548, offset=2424498}, end_cursor:ObLogCursor{file_id=1, log_id=8549, offset=2424795}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894680050, len:4430, log_data_len:297, seq:8548, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:48.982857] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB40-0-0] [lt=109][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193753896639, cache_obj->added_lc()=false, cache_obj->get_object_id()=13549, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:48.987293] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D2D-0-0] [lt=94] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:50:48.988984] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=51][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:42989337}], scn_array=[cnt:1, {val:1736326248971489000, v:0}], curr_size=750, accum_size=6524, time_cost=16602) [2025-01-08 00:50:48.990727] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58213][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB5B-0-0] [lt=26] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.138:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:48.990771] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58213][T1001_L0_G19][T1001][YB42C0A89F8A-00062B2D1D99BB5B-0-0] [lt=35] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.138:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:48.995147] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:48.995272] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=121][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:48.995341] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:48.995399] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:SHARED_BLOCK_ReaderWriter_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:55, bw:224145, [delay/us]:prepare:1, schedule:540, submit:0, rt:530, total:1073) [2025-01-08 00:50:48.995435] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4437, iops:12, bw:52979, [delay/us]:prepare:10, schedule:195, submit:72, rt:1768, total:2047) [2025-01-08 00:50:48.995454] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1, ref_cnt=3, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=49999, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=13, ops=3, iops=16, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:48.995540] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=73] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:55, avg_bw:224129, [delay/us]: prepare:1, schedule:540, submit:0, device:530, total:1073) [2025-01-08 00:50:48.995565] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4437, avg_iops:12, avg_bw:52975, [delay/us]: prepare:10, schedule:195, submit:72, device:1768, total:2047) [2025-01-08 00:50:48.995591] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:48.995635] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4577, iops:17, bw:77432, [delay/us]:prepare:7, schedule:185, submit:1068, rt:3045, total:4307) [2025-01-08 00:50:48.995657] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:48.995827] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=148] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4578, avg_iops:17, avg_bw:77422, [delay/us]: prepare:7, schedule:185, submit:1068, device:3045, total:4307) [2025-01-08 00:50:48.995854] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:48.995900] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:5120, iops:4, bw:20374, [delay/us]:prepare:4, schedule:53, submit:0, rt:2511, total:2569) [2025-01-08 00:50:48.995937] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4198, iops:40, bw:167074, [delay/us]:prepare:7, schedule:142, submit:656, rt:1998, total:2805) [2025-01-08 00:50:48.995958] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=11, iops=11, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:48.996122] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=154] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:5120, avg_iops:4, avg_bw:20371, [delay/us]: prepare:4, schedule:53, submit:0, device:2511, total:2569) [2025-01-08 00:50:48.996146] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4198, avg_iops:40, avg_bw:167048, [delay/us]: prepare:7, schedule:142, submit:656, device:1998, total:2805) [2025-01-08 00:50:48.996157] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:48.996179] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4074, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:303, total:312) [2025-01-08 00:50:48.996259] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:48.996297] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4074, [delay/us]: prepare:0, schedule:0, submit:600000000, device:303, total:312) [2025-01-08 00:50:48.996317] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:48.996333] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:48.996392] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=56] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:48.996401] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:48.996409] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=239kB/s, net_out=218kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:48.999990] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=76] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-01-08 00:50:49.001714] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=26] get number(ret=0, service_type_=0, range=1, base_id=1736326249001380000, start_id=1736326249001380000, end_id=1736326249001380001) [2025-01-08 00:50:49.008403] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=30] ====== checkpoint timer task ====== [2025-01-08 00:50:49.009276] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=45] succ to get rec scn(*this={this:0x2ad90c604030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324343298570, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:995}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:49.009395] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=117] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad90d098080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad90d098080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:49.009572] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=49] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324381352256003, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324381352256003, v:0}}, }, *this={this:0x2ad908da5150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324350282675001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736324353867845000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259284252000, latest_log_ts:{val:1736326242623339001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324350282675001, v:0}}, {limited_id:30000001, latest_log_ts:{val:1736324925494265001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:49.009838] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=260] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x2ad908da41b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x2ad908daf010, memtable_head:0, memtable_tail:1, t3m:0x2ad8fb604030, tables:[0x2ad90d088080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x2ad908db0d10, ls_tablet_svr:0x2ad908da4190}, min_rec_scn={val:1736324352538356001, v:0}, ls_id_={id:1}) [2025-01-08 00:50:49.009925] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=79] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:49.009957] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=29] get rec log scn(service_type_=0, rec_log_ts={val:1736326242623339001, v:0}) [2025-01-08 00:50:49.009969] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=10] get rec log scn(service_type_=1, rec_log_ts={val:1736324350282675001, v:0}) [2025-01-08 00:50:49.009976] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736324925494265001, v:0}) [2025-01-08 00:50:49.009988] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:49.010007] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:49.010063] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=52] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324350282675001, v:0}, checkpoint_scn_in_ls_meta={val:1736324350282675001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:49.010085] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:49.010107] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:49.010135] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=26] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:50:49.038145] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3C-0-0] [lt=128] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326249134834, keepalive_ts:-1}) [2025-01-08 00:50:49.038313] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3C-0-0] [lt=164] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326249134834, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:49.043315] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4597, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:49.043349] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=816, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:49.051476] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36A3-0-0] [lt=27] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=2390, avg_log_submit_to_slide_time=42664) [2025-01-08 00:50:49.051746] INFO [PALF] read (log_cache.cpp:103) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:49.051785] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17677, miss_cnt:0, cache_read_size:2162513, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:49.054815] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=44] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:49.062864] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:49.071570] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82DC-0-0] [lt=31] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=23, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:49.071613] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82DC-0-0] [lt=42] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=38181, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:49.071625] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82DC-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=25, avg_log_submit_to_flush_time=2560, avg_log_submit_to_slide_time=38156) [2025-01-08 00:50:49.072120] INFO [PALF] read (log_cache.cpp:103) [50478][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:49.079281] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=49] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:49.088663] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=18] success to update min reserved snapshot(reserved_snapshot=1736324444258401000, duration=1800, snapshot_gc_ts=1736326244258401000) [2025-01-08 00:50:49.093480] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=54] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324444258401000) [2025-01-08 00:50:49.093533] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=50] update reserved snapshot finished(cost_ts=4875, reserved_snapshot=1736324444258401000) [2025-01-08 00:50:49.095300] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.105295] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=36] re_register_parent reason(palf_id=1003, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326245095005, last_first_register_time_us=1736324454612308, register_time_us=1736326245095005, curr_time_us=1736326249105291) [2025-01-08 00:50:49.105362] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=36] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882", reason=1, leader="192.168.159.139:2882", register_time_us=1736326249105334, last_submit_register_req_time_us=1736326249105334, register_parent_reason=1) [2025-01-08 00:50:49.105380] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="192.168.159.137:2882") [2025-01-08 00:50:49.114515] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=99] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4076, avg_group_log_size=370) [2025-01-08 00:50:49.114547] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4076) [2025-01-08 00:50:49.114608] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:49.119520] INFO [COMMON] process (ob_rpc_processor_simple.cpp:2758) [58213][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAEC3-0-0] [lt=29] handle ObQueryLSIsValidMember requeset(arg_={tenant_id_:1001, self_addr_:"192.168.159.139:2882", ls_array_:[{id:1}]}) [2025-01-08 00:50:49.119581] INFO [CLOG] get_member_gc_stat (ob_log_handler.cpp:1619) [58213][T1001_L0_G19][T1001][YB42C0A89F8B-00062B2D1DABAEC3-0-0] [lt=52] get_member_gc_stat(is_valid_member=true, stat=1, member_list=3{server:"192.168.159.137:2882", timestamp:1, flag:0}{server:"192.168.159.138:2882", timestamp:1, flag:0}{server:"192.168.159.139:2882", timestamp:1, flag:0}, learner_list={learner_num:0, learner_array:[]}, addr="192.168.159.139:2882", this={role:1, proposal_id:1, palf_env_:0x2ad8f97e8030, is_in_stop_state_:false, is_inited_:true, id_:1}) [2025-01-08 00:50:49.124685] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:49.126356] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=51] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=346027, clean_num=31457) [2025-01-08 00:50:49.145596] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=49] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:49.149895] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [46343][T1_TenantWeakRe][T1][YB42C0A89F89-00062B2D19690D1B-0-0] [lt=33] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=1.794224309921264648e+02, this=0x2ad8955ed208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=1023, Resource Limit(MB)=897, Throttle Trigger(MB)=538, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=2.748260800254538339e+01) [2025-01-08 00:50:49.150764] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=27] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:1408K/347:176556 disk:1408K/347:176556 t1m2:1140K/284:172755 t1m2g0:1140K/284:172755 t1002m2:184K/44:155021 t1002m2g0:184K/44:155021 t1001m2:84K/19:175093 t1001m2g0:84K/19:175093 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:49.154279] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=57] get number(ret=0, service_type_=0, range=1, base_id=1736326249153254000, start_id=1736326249153254000, end_id=1736326249153254001) [2025-01-08 00:50:49.154450] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT APPEND COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:49.155345] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25] ====== check clog disk timer task ====== [2025-01-08 00:50:49.155373] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=279) [2025-01-08 00:50:49.160340] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:50:49.162508] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:93) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=30] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-01-08 00:50:49.164373] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326249162225000, start_id=1736326249162225000, end_id=1736326249162225001) [2025-01-08 00:50:49.170388] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=18641702, this={dio_aligned_buf:{buf_write_offset:1351, buf_padding_size:2745, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:32013, truncate_used_ts:6657}, log_block_size:67108864, io_fd:{first_id:-1, second_id:176, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=18640896, buf_len=545, total_write_size=185050, total_write_size_after_dio=696320, ob_pwrite_used_ts=358235, count=125, each_pwrite_cost=2865) [2025-01-08 00:50:49.175430] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1C8BC365-0-0] [lt=28][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.195395] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.195436] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB42-0-0] [lt=21][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193754109267, cache_obj->added_lc()=false, cache_obj->get_object_id()=13550, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:49.214660] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=16] gc stale ls task succ [2025-01-08 00:50:49.220127] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:49.220181] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:49.223349] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87007-0-0] [lt=14][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.223395] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:49.223406] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:49.246090] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=45] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:49.246116] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=25] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:50:49.246239] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E0-0-0] [lt=99] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:49.246282] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E0-0-0] [lt=40] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:49.258335] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=26] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:49.277740] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C97-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:49.277778] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C97-0-0] [lt=35] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:49.277815] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:50:49.277860] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:49.277880] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] temporary file module use no blocks [2025-01-08 00:50:49.277888] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=6] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:49.277901] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:49.277915] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193753188851, flush_io_finished_ret_:0, flush_io_finished_round_:1912, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:49.289382] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540C-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.295426] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.295957] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:750) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=8] finish to resize log external storage handler(current_ts=1736326249295952, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-01-08 00:50:49.295992] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:241) [46322][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=32] snapshot_traffic_info once [2025-01-08 00:50:49.308349] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58207][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBBC-0-0] [lt=86] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326249405156, keepalive_ts:-1}) [2025-01-08 00:50:49.308420] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58207][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CC9BBBC-0-0] [lt=70] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326249405156, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:49.311116] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=29] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-01-08 00:50:49.311163] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=41] start lib cache evict(tenant_id=1002, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:50:49.311361] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=194] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=102760448, mem_limit=161061270, cache_obj_num=329, cache_node_num=327) [2025-01-08 00:50:49.311376] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:49.314079] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=329, alloc_obj_list=[{obj_id:184, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179800}, {obj_id:185, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179056}, {obj_id:186, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:187, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:189, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184800}, {obj_id:191, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:194, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:214, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:215, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185324}, {obj_id:216, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189554}, {obj_id:217, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:218, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:299808}, {obj_id:220, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228085}, {obj_id:222, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215292}, {obj_id:225, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180521}, {obj_id:228, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178402}, {obj_id:234, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:453298}, {obj_id:292, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177072}, {obj_id:336, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264566}, {obj_id:357, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2291787}, {obj_id:358, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:2648585}, {obj_id:359, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222912}, {obj_id:360, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223297}, {obj_id:361, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213594}, {obj_id:362, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215737}, {obj_id:363, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239696}, {obj_id:364, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241758}, {obj_id:365, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240683}, {obj_id:366, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:532933}, {obj_id:367, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191035}, {obj_id:368, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:341395}, {obj_id:369, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:343530}, {obj_id:370, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197644}, {obj_id:371, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204301}, {obj_id:372, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204905}, {obj_id:373, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179422}, {obj_id:374, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207400}, {obj_id:378, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:327198}, {obj_id:379, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258520}, {obj_id:380, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:312672}, {obj_id:381, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243991}, {obj_id:382, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180416}, {obj_id:383, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184298}, {obj_id:384, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179712}, {obj_id:385, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185454}, {obj_id:386, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236669}, {obj_id:387, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192151}, {obj_id:388, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201094}, {obj_id:389, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177880}, {obj_id:390, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323762}, {obj_id:391, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228134}, {obj_id:392, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234205}, {obj_id:394, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207110}, {obj_id:395, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213189}, {obj_id:396, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207208}, {obj_id:397, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203337}, {obj_id:398, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197116}, {obj_id:399, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220151}, {obj_id:400, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}, {obj_id:401, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208334}, {obj_id:402, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194033}, {obj_id:403, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190716}, {obj_id:404, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189404}, {obj_id:405, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204465}, {obj_id:406, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198372}, {obj_id:407, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192279}, {obj_id:408, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189580}, {obj_id:409, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192620}, {obj_id:410, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189612}, {obj_id:411, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192621}, {obj_id:412, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189548}, {obj_id:413, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196031}, {obj_id:414, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191118}, {obj_id:415, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187276}, {obj_id:416, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214929}, {obj_id:417, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187052}, {obj_id:418, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192050}, {obj_id:419, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187452}, {obj_id:420, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199077}, {obj_id:421, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195907}, {obj_id:422, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187148}, {obj_id:424, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:425, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188348}, {obj_id:426, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195840}, {obj_id:427, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:428, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190558}, {obj_id:429, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:430, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:332235}, {obj_id:431, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:432, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:433, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:434, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:262127}, {obj_id:435, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307894}, {obj_id:436, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237798}, {obj_id:437, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178056}, {obj_id:439, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:302729}, {obj_id:440, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:265482}, {obj_id:441, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177928}, {obj_id:442, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193473}, {obj_id:443, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:444, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:445, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:446, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:447, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:448, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248104}, {obj_id:449, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:450, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:451, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:452, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:453, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:281790}, {obj_id:454, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244536}, {obj_id:457, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182896}, {obj_id:459, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210098}, {obj_id:461, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185689}, {obj_id:462, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180056}, {obj_id:463, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190600}, {obj_id:464, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176840}, {obj_id:465, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:273516}, {obj_id:466, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:467, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:468, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:469, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253385}, {obj_id:470, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252993}, {obj_id:471, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222840}, {obj_id:472, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222457}, {obj_id:473, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193841}, {obj_id:474, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191704}, {obj_id:475, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188352}, {obj_id:476, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201060}, {obj_id:477, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193313}, {obj_id:478, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195665}, {obj_id:479, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186441}, {obj_id:481, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184843}, {obj_id:482, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184082}, {obj_id:483, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192248}, {obj_id:484, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189040}, {obj_id:485, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185817}, {obj_id:486, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184265}, {obj_id:487, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186165}, {obj_id:488, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184297}, {obj_id:489, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186166}, {obj_id:491, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184239}, {obj_id:493, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187868}, {obj_id:494, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185241}, {obj_id:495, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182865}, {obj_id:496, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198106}, {obj_id:497, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182881}, {obj_id:498, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185593}, {obj_id:499, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183289}, {obj_id:500, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187720}, {obj_id:501, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182976}, {obj_id:502, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183609}, {obj_id:503, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188691}, {obj_id:504, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184663}, {obj_id:505, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:307782}, {obj_id:506, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268366}, {obj_id:507, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272110}, {obj_id:508, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232692}, {obj_id:509, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188740}, {obj_id:510, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184389}, {obj_id:511, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195391}, {obj_id:512, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:513, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186202}, {obj_id:514, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185413}, {obj_id:515, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176688}, {obj_id:516, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:517, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184937}, {obj_id:518, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:519, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:520, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185024}, {obj_id:521, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192449}, {obj_id:522, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190171}, {obj_id:523, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189328}, {obj_id:524, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:525, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:526, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248264}, {obj_id:527, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:528, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:529, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195949}, {obj_id:530, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176624}, {obj_id:531, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:532, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189141}, {obj_id:533, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190297}, {obj_id:534, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198826}, {obj_id:535, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235142}, {obj_id:536, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:537, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179904}, {obj_id:538, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:540, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:541, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:542, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202042}, {obj_id:543, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183342}, {obj_id:544, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:545, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193268}, {obj_id:546, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:547, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:548, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189760}, {obj_id:549, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178480}, {obj_id:550, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183097}, {obj_id:551, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:553, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188615}, {obj_id:554, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:555, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239800}, {obj_id:556, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241751}, {obj_id:557, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:558, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181131}, {obj_id:559, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:560, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:568, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:570, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:571, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195541}, {obj_id:577, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181874}, {obj_id:579, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188393}, {obj_id:589, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:320840}, {obj_id:592, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:593, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:260287}, {obj_id:594, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:595, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:297040}, {obj_id:598, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:229083}, {obj_id:600, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236949}, {obj_id:601, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189628}, {obj_id:602, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190140}, {obj_id:604, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:14054}, {obj_id:605, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182995}, {obj_id:606, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191920}, {obj_id:607, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1234612}, {obj_id:608, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186684}, {obj_id:609, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:975017}, {obj_id:610, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:611, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:252367}, {obj_id:614, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233751}, {obj_id:615, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:616, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:617, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:618, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:619, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:620, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:621, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209724}, {obj_id:622, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:381007}, {obj_id:623, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189994}, {obj_id:624, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185692}, {obj_id:625, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183053}, {obj_id:627, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:268772}, {obj_id:628, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344835}, {obj_id:629, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1027854}, {obj_id:630, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176557}, {obj_id:631, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178152}, {obj_id:632, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279851}, {obj_id:633, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214505}, {obj_id:634, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1038892}, {obj_id:635, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202121}, {obj_id:636, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:637, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235899}, {obj_id:638, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:639, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235514}, {obj_id:641, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246914}, {obj_id:642, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:562543}, {obj_id:643, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193399}, {obj_id:644, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552398}, {obj_id:645, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:236395}, {obj_id:646, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:336848}, {obj_id:647, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189221}, {obj_id:649, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209553}, {obj_id:650, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:383957}, {obj_id:651, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185387}, {obj_id:652, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:366701}, {obj_id:653, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328818}, {obj_id:655, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346943}, {obj_id:656, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1036694}, {obj_id:657, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214841}, {obj_id:658, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1048084}, {obj_id:659, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316807}, {obj_id:661, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248858}, {obj_id:662, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:571156}, {obj_id:663, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193797}, {obj_id:664, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:561379}, {obj_id:665, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:666, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237399}, {obj_id:667, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:263658}, {obj_id:668, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183538}, {obj_id:669, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233226}, {obj_id:670, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187396}, {obj_id:671, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180726}, {obj_id:672, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:222665}, {obj_id:673, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179152}, {obj_id:674, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:264475}, {obj_id:675, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230312}, {obj_id:676, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:300633}, {obj_id:677, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185909}, {obj_id:678, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188616}, {obj_id:679, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239768}, {obj_id:680, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241719}, {obj_id:681, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411588}, {obj_id:682, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212962}, {obj_id:683, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289845}, {obj_id:684, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:434174}, {obj_id:686, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:328071}, {obj_id:687, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:451872}, {obj_id:688, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:370098}, {obj_id:689, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206700}, {obj_id:690, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192229}, {obj_id:691, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283899}, {obj_id:692, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293641}, {obj_id:693, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:293642}, {obj_id:694, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223530}, {obj_id:695, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185745}, {obj_id:696, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242589}, {obj_id:697, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:242590}, {obj_id:698, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203730}, {obj_id:727, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177736}, {obj_id:728, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178464}, {obj_id:838, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209284}, {obj_id:839, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186847}, {obj_id:840, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211235}, {obj_id:841, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185349}, {obj_id:842, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185285}, {obj_id:844, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184022}, {obj_id:846, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:593755}, {obj_id:847, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272530}, {obj_id:848, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239335}, {obj_id:849, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:851, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:852, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:853, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:854, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:855, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:856, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:207152}, {obj_id:857, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240136}, {obj_id:858, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}]) [2025-01-08 00:50:49.315120] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=1036] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:49.319323] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.319365] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.327130] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:49.329435] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2025-01-08 00:50:49.336160] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E3-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:49.336303] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E3-0-0] [lt=124] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326249201328000, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326249201328000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:49.341178] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=73, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:49.342706] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=33] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:50:49.353162] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36A9-0-0] [lt=26] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:49.353583] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=28] gc stale ls task succ [2025-01-08 00:50:49.356249] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=31] [PALF STAT WRITE LOG COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=4706, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:49.356395] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=144] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9781) [2025-01-08 00:50:49.356430] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=239, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:49.356506] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=20, stat_interval=1000000, avg (count/cost)=29, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:49.357475] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58207][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B3D-0-0] [lt=17] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=20, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=109, avg_log_submit_to_flush_time=5095, avg_log_submit_to_slide_time=5880) [2025-01-08 00:50:49.357600] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=29] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9781, hit_cnt=21, read_cnt=21, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:49.358646] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326248700933003, v:0}, new_max_version={val:1736326248700933003, v:0}, last_min_version={val:1736326248488325003, v:0}, last_max_version={val:1736326248488325003, v:0}, total_time_us=6200, delta_us=657711) [2025-01-08 00:50:49.358679] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326248700933003, v:0}, min_version={val:1736326248700933003, v:0}, max_version={val:1736326248700933003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:49.360631] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:50:49.361042] INFO [LIB] stat (utility.h:1210) [58307][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37D6-0-0] [lt=84] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=19, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:49.361064] INFO [LIB] stat (utility.h:1210) [58307][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37D6-0-0] [lt=22] [PALF STAT LOG LIFE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=5884, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:49.361471] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=24] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:49.365920] INFO [STORAGE] find_adaptive_merge_tables (ob_partition_merge_policy.cpp:1601) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8574F-0-0] [lt=40] no enough table or no enough rows for meta merge(ret=-4677, scanty_tx_determ_table=false, scanty_inc_row_cnt=true, inc_row_cnt=9, base_inc_row_cnt=0, result={version_range:{multi_version_start:1736322630524987000, base_version:0, snapshot_version:1736322630524987000}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1736322632627436002, v:0}}, merge_version:0, is_simplified:false, handle:{tablet_id:{id:200001}, table_count:2, [{i:0, table_key:{tablet_id:{id:200001}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1, v:0}}}, ref:0}{i:1, table_key:{tablet_id:{id:200001}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:1736322632627436002, v:0}}}, ref:0}]}, update_tablet_directly:false, schedule_major:false, is_backfill:false, backfill_scn:{val:18446744073709551615, v:3}, transfer_seq:-1}, PRINT_TS_WRAPPER(table_store_wrapper)={ObTabletTableStore_Pretty:this:0x2ad8a9350838, major_tables:{cnt:1, sstable_array:0x2ad8ebbea890, serialize_table_type:false, is_inited:true}, minor_tables:{cnt:1, sstable_array:0x2ad8ebbeac60, serialize_table_type:false, is_inited:true}, mds_sstables:{cnt:1, sstable_array:0x2ad8ebbeb686, serialize_table_type:false, is_inited:true}, memtables:{count:0}, is_ready_for_read:true, table_array:{ [46307][T1_TimerWK3][T1] [YB42C0A89F89-00062B2D16E8574F-0-0] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [46307][T1_TimerWK3][T1] [YB42C0A89F89-00062B2D16E8574F-0-0] MAJOR MAJOR 0 1 0 1 0 false [46307][T1_TimerWK3][T1] [YB42C0A89F89-00062B2D16E8574F-0-0] MINOR MINI 1736322630524987000 1736322630524987000 1 1736322632627436002 0 false [46307][T1_TimerWK3][T1] [YB42C0A89F89-00062B2D16E8574F-0-0] MDS MDS_MINI 1736322603189421000 1736322603189421000 1 1736322605489896000 0 false }}) [2025-01-08 00:50:49.372522] INFO [CLOG] handle_submit_task_ (ob_log_replay_service.cpp:1291) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] succ to submit log task to replay service(to_submit_lsn={lsn:2173513}, to_submit_scn={val:1736326248991181003, v:0}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326249372377, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2173579}, next_to_submit_scn_:{val:1736326248991181004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2173457}, end_lsn:{lsn:2173579}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2173579}, readable_log_tail:{lsn:2173579}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64931189, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326248991181003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326248991181003, v:0}, data_checksum:3348837161, flag:27546}}, init_mode_version:0, accumulate_checksum:988638107, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326248991181003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17769, miss_cnt:0, cache_read_size:2173579, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:49.389483] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=141] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:49.389524] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-01-08 00:50:49.389534] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:49.389592] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=54] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:49.389606] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193753303140, flush_io_finished_ret_:0, flush_io_finished_round_:1908, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:49.392945] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=95] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:49.393109] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=158] temporary file module use no blocks [2025-01-08 00:50:49.393119] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:49.393131] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:49.393143] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193753303554, flush_io_finished_ret_:0, flush_io_finished_round_:3736, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:49.395514] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.404925] INFO [STORAGE.TRANS] alloc (ob_ts_response_handler.cpp:108) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=80] ts response task statistics(alloc_count=59382, free_count=29690) [2025-01-08 00:50:49.407092] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [46329][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=23] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:49.418574] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [50444][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=41] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-01-08 00:50:49.435565] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:775) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] success to update min reserved snapshot(reserved_snapshot=1736324445564663000, duration=1800, snapshot_gc_ts=1736326245564663000) [2025-01-08 00:50:49.438714] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=44] submit reserved snapshot log success(ls_id={id:1}, new_snapshot_version=1736324445564663000) [2025-01-08 00:50:49.439708] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12] ====== tenant freeze timer task ====== [2025-01-08 00:50:49.439741] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70285300, tenant_id=1001) [2025-01-08 00:50:49.439763] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:50:49.443735] INFO [STORAGE] update_min_reserved_snapshot_for_leader (ob_ls_reserved_snapshot_mgr.cpp:191) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=27] submit reserved snapshot log success(ls_id={id:1001}, new_snapshot_version=1736324445564663000) [2025-01-08 00:50:49.443768] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:802) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=29] update reserved snapshot finished(cost_ts=8208, reserved_snapshot=1736324445564663000) [2025-01-08 00:50:49.446122] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F2E-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=27, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:49.446160] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F2E-0-0] [lt=37] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=6769, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:49.446453] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=126] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1755, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:49.451545] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=302] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:49.452878] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:49.453533] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36AB-0-0] [lt=11] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1333, avg_group_log_size=121) [2025-01-08 00:50:49.453687] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36AB-0-0] [lt=139] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=10, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:49.453701] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36AB-0-0] [lt=13] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=47522, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:49.453710] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58289][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36AB-0-0] [lt=7][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17770, log_task={header:{begin_lsn:{lsn:2162879}, end_lsn:{lsn:2163001}, log_id:17770, min_scn:{val:1736326249201328001, v:0}, max_scn:{val:1736326249201328001, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2162757}, prev_proposal_id:1, committed_end_lsn:{lsn:2162879}, data_checksum:940009244, accum_checksum:2769181528, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326249353154, freeze_ts:1736326249353155, submit_ts:1736326249353260, flushed_ts:1736326249359124, gen_to_freeze cost time:1, gen_to_submit cost time:106, submit_to_flush cost time:5864}, fs_cb_begin_ts=1736326249453679, log_life_time=100525) [2025-01-08 00:50:49.459414] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=34] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.459415] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=75] trace max work area(auto_calc=true, tenant_max_memory_limit=1073741824, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=53687091, max_wa_memory_size=53687091, tmp_max_wa_memory_size=53687091, pre_mem_target=51589939, remain_memory_size=53687091, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=656101376, washable_size=-2, max_workarea_memory_size=53687091, max_tenant_memory_size=417640448, wash_ratio=6, tenant_id=1001) [2025-01-08 00:50:49.459543] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=38] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=6710886, calc_global_bound_size=1048576000000, mem_target_=53687091, wa_max_memory_size=53687091, profile_cnt_=0, total_memory_size=0, max_wa_size=53687091, avg_bound_size=6710886, max_bound_size=6710886, min_bound_size_=1048576) [2025-01-08 00:50:49.459625] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=78] timer to calc global bound size(ret=0, global_bound_size_=6710886, manual_calc_cnt_=2790, drift_size_=0, pre_drift_size=-8497152, wa_max_memory_size=53687091, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1001, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=6710886, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=53687091, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:49.459694] INFO [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:944) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=17] trace max work area(auto_calc=true, tenant_max_memory_limit=3221225472, total_alloc_size=0, tenant_work_area_memory_hold=0, tenant_work_area_max_size=161061273, max_wa_memory_size=161061273, tmp_max_wa_memory_size=161061273, pre_mem_target=161061273, remain_memory_size=161061273, ratio=-nan, alloc_ratio=0.000000000000000000e+00, hold_ratio=0.000000000000000000e+00, tenant_memory_hold=1110196224, washable_size=-2, max_workarea_memory_size=161061273, max_tenant_memory_size=2111029248, wash_ratio=6, tenant_id=1002) [2025-01-08 00:50:49.459750] INFO [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:310) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=20] timer to calc global bound size(ret=0, best_interval_idx=1099, global_bound_size_=20132659, calc_global_bound_size=1048576000000, mem_target_=161061273, wa_max_memory_size=161061273, profile_cnt_=0, total_memory_size=0, max_wa_size=161061273, avg_bound_size=20132659, max_bound_size=20132659, min_bound_size_=1048576) [2025-01-08 00:50:49.459764] INFO [SQL.ENG] calculate_global_bound_size_by_interval_info (ob_tenant_sql_memory_manager.cpp:1192) [46101][TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] timer to calc global bound size(ret=0, global_bound_size_=20132659, manual_calc_cnt_=23, drift_size_=0, pre_drift_size=0, wa_max_memory_size=161061273, sql_mem_callback_.get_total_alloc_size()=0, tenant_id_=1002, profile_cnt_=0, pre_profile_cnt_=0, pre_profile_cnt=0, calc_info.get_global_bound_size()=20132659, total_memory_size=0, cur_profile_cnt=0, calc_info.get_mem_target()=161061273, auto_calc=true, sql_mem_callback_.get_total_dump_size()=0) [2025-01-08 00:50:49.459870] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.468693] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-01-08 00:50:49.469054] INFO [LIB] stat (utility.h:1210) [58289][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43E9-0-0] [lt=31] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:49.470653] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=65] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:49.471890] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58289][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=248] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:50:49.483532] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=81] PNIO [ratelimit] time: 1736326249483529, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007629, add_bytes: 0 [2025-01-08 00:50:49.495694] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.517329] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=64] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1923, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:49.517384] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=53] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=4076) [2025-01-08 00:50:49.517412] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=19] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=78, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:49.517449] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=40] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=18, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:49.519512] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A7F-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=25, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:49.519549] INFO [LIB] stat (utility.h:1210) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A7F-0-0] [lt=36] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4208, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:49.519563] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58301][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A7F-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=10, avg_log_gen_to_submit_time=94, avg_log_submit_to_flush_time=2177, avg_log_submit_to_slide_time=4113) [2025-01-08 00:50:49.519793] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=4076, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:49.519831] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=36] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.527624] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:49.529633] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=21, avg_log_batch_cnt=1, total_group_log_size=10530, avg_group_log_size=501) [2025-01-08 00:50:49.529662] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=24] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10530) [2025-01-08 00:50:49.541623] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=76] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=40, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:49.541671] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:50:49.541692] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1332) [2025-01-08 00:50:49.541706] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:49.542056] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=188] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.542516] INFO pn_ratelimit (group.c:653) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=46] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-01-08 00:50:49.542683] INFO [STORAGE] try_shrink (ob_compaction_memory_pool.cpp:522) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=39] Compaction Mem Pool current stat: (ret=0, max_block_num=64, total_block_num=8, Chunk list size: =1) [2025-01-08 00:50:49.544242] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1332) [2025-01-08 00:50:49.544506] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.545352] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC82EB-0-0] [lt=53] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=6, avg_log_gen_to_submit_time=97, avg_log_submit_to_flush_time=5319, avg_log_submit_to_slide_time=6495) [2025-01-08 00:50:49.545566] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F32-0-0] [lt=117] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=24, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:49.545812] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F32-0-0] [lt=244] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4716, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:49.548803] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=132] no need to update min/max version(new_version={val:1736326249147402000, v:0}, min_version={val:1736326249147402000, v:0}, max_version={val:1736326249147402000, v:0}) [2025-01-08 00:50:49.548832] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:49.554663] INFO [STORAGE.TRANS] alloc (ob_trans_factory.cpp:141) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ABD5-0-0] [lt=158] ObTransCtx statistics(active_coord_ctx_count=0, active_part_ctx_count=1, total_release_part_ctx_count=108) [2025-01-08 00:50:49.555515] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:49.557903] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=17] [APPLY STAT CB APPEND COST TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=152, this=0x2ad90d06a280, extra_info="") [2025-01-08 00:50:49.557928] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=25] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=5939, this=0x2ad90d06a4b0, extra_info="") [2025-01-08 00:50:49.557937] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=0, this=0x2ad90d06a6e0, extra_info="") [2025-01-08 00:50:49.557945] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=8] [APPLY STAT CB EXECUTE TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=-1045, this=0x2ad90d06a910, extra_info="") [2025-01-08 00:50:49.557953] INFO [LIB] stat (utility.h:1210) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=74, stat_interval=5000000, avg (count/cost)=5046, this=0x2ad90d06ab40, extra_info="") [2025-01-08 00:50:49.561032] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:50:49.563520] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:49.579634] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:49.595950] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.621491] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=15] [APPLY STAT CB APPEND COST TIME](cur_stat_count=37, stat_interval=5000000, avg (count/cost)=156, this=0x2ad8aed4c280, extra_info="") [2025-01-08 00:50:49.621528] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=37] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=37, stat_interval=5000000, avg (count/cost)=3970, this=0x2ad8aed4c4b0, extra_info="") [2025-01-08 00:50:49.621537] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=37, stat_interval=5000000, avg (count/cost)=0, this=0x2ad8aed4c6e0, extra_info="") [2025-01-08 00:50:49.621698] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=159] [APPLY STAT CB EXECUTE TIME](cur_stat_count=37, stat_interval=5000000, avg (count/cost)=-779, this=0x2ad8aed4c910, extra_info="") [2025-01-08 00:50:49.621707] INFO [LIB] stat (utility.h:1210) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=9] [APPLY STAT CB TOTAL TIME](cur_stat_count=37, stat_interval=5000000, avg (count/cost)=3347, this=0x2ad8aed4cb40, extra_info="") [2025-01-08 00:50:49.643060] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=205] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3198, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:49.643125] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=47] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=269, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:49.646622] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=6] [APPLY STAT CB APPEND COST TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=124, this=0x2ad934926280, extra_info="") [2025-01-08 00:50:49.646665] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=43] [APPLY STAT CB IN QUEUE TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=5538, this=0x2ad9349264b0, extra_info="") [2025-01-08 00:50:49.646677] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=11] [APPLY STAT CB WAIT COMMIT TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=0, this=0x2ad9349266e0, extra_info="") [2025-01-08 00:50:49.646685] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB EXECUTE TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=-784, this=0x2ad934926910, extra_info="") [2025-01-08 00:50:49.646692] INFO [LIB] stat (utility.h:1210) [50433][T1002_ApplySrv0][T1002][Y0-0000000000000000-0-0] [lt=7] [APPLY STAT CB TOTAL TIME](cur_stat_count=23, stat_interval=5000000, avg (count/cost)=4879, this=0x2ad934926b40, extra_info="") [2025-01-08 00:50:49.646861] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:49.651478] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36AF-0-0] [lt=12][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17772, log_task={header:{begin_lsn:{lsn:2163123}, end_lsn:{lsn:2163245}, log_id:17772, min_scn:{val:1736326249201328003, v:0}, max_scn:{val:1736326249201328003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2163001}, prev_proposal_id:1, committed_end_lsn:{lsn:2163123}, data_checksum:940009244, accum_checksum:1691600049, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326249551452, freeze_ts:1736326249551452, submit_ts:1736326249551461, flushed_ts:1736326249555884, gen_to_freeze cost time:0, gen_to_submit cost time:9, submit_to_flush cost time:4423}, fs_cb_begin_ts=1736326249651468, log_life_time=100016) [2025-01-08 00:50:49.661872] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=90] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:50:49.666225] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58301][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=13] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:50:49.668868] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43ED-0-0] [lt=75] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:49.670275] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1558, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:49.670317] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:50:49.670341] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=128, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:49.670403] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=61, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:49.691140] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=12] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:50:49.696103] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.728395] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=113] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:49.728735] INFO [COMMON] wash (ob_kvcache_store.cpp:356) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=48] Wash time detail, (compute_wash_size_time=147, refresh_score_time=49, wash_time=0) [2025-01-08 00:50:49.756524] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2750, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:49.756563] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=308, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:49.756653] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=45, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:49.761534] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:50:49.766291] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=31] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:49.769418] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43EF-0-0] [lt=103] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1342, avg_group_log_size=122) [2025-01-08 00:50:49.773388] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=90, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:49.773430] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=13, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:49.774933] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:49.790403] INFO [PALF] is_allowed_broadcast_ (palf_handle_impl.h:1404) [50381][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=92] keep in silent because of disconnected with RS, do not solicit votes.(is_in_silent=false, palf_id=1001, is_allowed_broadcast=true) [2025-01-08 00:50:49.796225] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.800623] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58299][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B45-0-0] [lt=35] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10537) [2025-01-08 00:50:49.805607] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=38] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:49.809061] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50624][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1C49CFFF-0-0] [lt=42][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.816673] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=77, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:49.834591] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1736326249834587, bytes: 866070813, bw: 0.218721 MB/s, add_ts: 1004177, add_bytes: 230304 [2025-01-08 00:50:49.844161] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1755) [2025-01-08 00:50:49.848565] INFO [STORAGE.COMPACTION] add_event_and_diagnose (ob_medium_loop.cpp:203) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8574F-0-0] [lt=100] finish schedule all tablet merge(merge_version_=1, schedule_stats_={all_ls_weak_read_ts_ready:true, start_timestamp:0}, loop_cnt=0, tenant_merge_finish=true, is_scan_all_tablet_finish=true, schedule_tablet_cnt={schedule_dag_cnt:0, submit_clog_cnt:0, finish_cnt:1105, loop_tablet_cnt:1105, force_freeze_cnt:0, wait_rs_validate_cnt:0}, time_guard=GET_TABLET=20.27ms|INIT_TABLET_STATUS=89.54ms|SCHEDULE_NEXT_MEDIUM=7.62ms|SCHEDULE_TABLET_EXECUTE=819.37ms|) [2025-01-08 00:50:49.848620] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:95) [46307][T1_TimerWK3][T1][YB42C0A89F89-00062B2D16E8574F-0-0] [lt=49] MediumLoopTask(cost_ts=937129) [2025-01-08 00:50:49.851827] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5180384760, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:49.851863] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:49.851876] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8bf5f0530) [2025-01-08 00:50:49.851885] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5180384760, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:49.855828] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C81-0-0] [lt=48][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.856444] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1001}, {id:1}, {id:1002}]) [2025-01-08 00:50:49.857719] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:167) [46304][T1_TimerWK0][T1][YB42C0A89F89-00062B2D16F85790-0-0] [lt=47] MediumCheckTask(cost_ts=8) [2025-01-08 00:50:49.858049] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1342) [2025-01-08 00:50:49.864518] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C81-0-0] [lt=45][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:49.888760] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=80] ====== checkpoint timer task ====== [2025-01-08 00:50:49.889867] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=74] succ to get rec scn(*this={this:0x2ad8afe04030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736322513734723, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1111}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:49.890051] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=181] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8aed7a080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8aed7a080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:49.890301] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=115] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736322631884291001, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:1152921504606846995}, rec_scn:{val:1736322631884291001, v:0}}, }, *this={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259254996000, latest_log_ts:{val:1736326242681587001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:49.890484] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=176] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8aec7b150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736322549262140001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1736322615522513000, v:0}, all_id_meta:{id_meta:[{limited_id:1736326259254996000, latest_log_ts:{val:1736326242681587001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736322549262140001, v:0}}, {limited_id:20000001, latest_log_ts:{val:1736322551426268003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8a09e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}, min_rec_scn={val:1736322631884291001, v:0}, ls_id_={id:1}) [2025-01-08 00:50:49.890637] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=143] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:49.890653] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:1736326242681587001, v:0}) [2025-01-08 00:50:49.890662] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=1, rec_log_ts={val:1736322549262140001, v:0}) [2025-01-08 00:50:49.890701] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:1736322551426268003, v:0}) [2025-01-08 00:50:49.890771] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=57] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:49.890855] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=64] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:49.890892] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=30] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736322549262140001, v:0}, checkpoint_scn_in_ls_meta={val:1736322549262140001, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:49.890930] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:49.890977] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=45] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:49.890992] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=11] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-01-08 00:50:49.896592] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.897348] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=41] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.897406] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.897476] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=113] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.897509] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:49.905831] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:49.929393] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50473][T1002_DBMSSched][T1002][YB42C0A89F89-00062B2D22584A88-0-0] [lt=25] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:49.929718] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:49.930365] INFO [LIB] stat (utility.h:1210) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=255] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=21, stat_interval=1000000, avg (count/cost)=93, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:49.942389] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:49.946663] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:411) [46105][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=151] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=12816728064, fast_delete_log_mode=false) [2025-01-08 00:50:49.946747] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=7] ps cache evict(tenant_id=1, stmt_id_map_.size()=10, stmt_info_map_.size()=10, op.get_used_size()=23327, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=10) [2025-01-08 00:50:49.947152] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1332) [2025-01-08 00:50:49.947401] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=26, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:49.947577] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F3E-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1877) [2025-01-08 00:50:49.951868] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:49.953286] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:49.954073] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=97] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:49.954498] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58207][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=25] worker thread exit(tenant_->id()=1001, workers_.get_size()=5) [2025-01-08 00:50:49.955136] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58337][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:49.957437] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=163] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:50:49.961694] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:50:49.973504] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58337][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D600-0-0] [lt=27] init point thread id with(&point=0x5572d50e2180, idx_=993, point=[thread id=58337, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58337) [2025-01-08 00:50:49.973659] INFO [PALF] next (palf_iterator.h:183) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17775, miss_cnt:0, cache_read_size:2174311, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:49.980597] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=68] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8549, offset=2424795}, end_cursor:ObLogCursor{file_id=1, log_id=8550, offset=2425100}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894684050, len:4133, log_data_len:305, seq:8549, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:49.981029] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3B-0-0] [lt=27] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:49.981078] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3B-0-0] [lt=27] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326249904148000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326249904148000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:49.993015] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=49] get number(ret=0, service_type_=0, range=1, base_id=1736326249989823000, start_id=1736326249989823000, end_id=1736326249989823001) [2025-01-08 00:50:49.995566] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=37] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8550, offset=2425100}, end_cursor:ObLogCursor{file_id=1, log_id=8551, offset=2425397}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894688050, len:3828, log_data_len:297, seq:8550, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:49.996200] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3B-0-0] [lt=25] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:49.996295] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3B-0-0] [lt=73] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326249904148000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326249904148000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:49.996871] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:49.998389] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:49.998433] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:SHARED_BLOCK_ReaderWriter_IO, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:1047, bw:4287769, [delay/us]:prepare:1, schedule:189, submit:0, rt:350, total:541) [2025-01-08 00:50:49.998457] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4551, iops:9, bw:40835, [delay/us]:prepare:11, schedule:422, submit:131, rt:1145, total:1709) [2025-01-08 00:50:49.998524] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=58] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=261, ops=2, iops=263, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:49.998618] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=87] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1047, avg_bw:4287585, [delay/us]: prepare:1, schedule:189, submit:0, device:350, total:541) [2025-01-08 00:50:49.998637] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4551, avg_iops:9, avg_bw:40834, [delay/us]: prepare:11, schedule:422, submit:131, device:1145, total:1709) [2025-01-08 00:50:49.998649] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:49.998690] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4527, iops:19, bw:85754, [delay/us]:prepare:7, schedule:762, submit:348, rt:1672, total:2791) [2025-01-08 00:50:49.998701] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:49.998739] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4527, avg_iops:19, avg_bw:85765, [delay/us]: prepare:7, schedule:762, submit:348, device:1672, total:2791) [2025-01-08 00:50:49.998750] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:49.998771] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:6144, iops:2, bw:12252, [delay/us]:prepare:7, schedule:131, submit:1, rt:1148, total:1288) [2025-01-08 00:50:49.998793] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4193, iops:42, bw:175622, [delay/us]:prepare:6, schedule:344, submit:184, rt:1466, total:2002) [2025-01-08 00:50:49.998802] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:49.998879] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=71] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:6144, avg_iops:2, avg_bw:12254, [delay/us]: prepare:7, schedule:131, submit:1, device:1148, total:1288) [2025-01-08 00:50:49.998896] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4194, avg_iops:42, avg_bw:175644, [delay/us]: prepare:6, schedule:344, submit:184, device:1466, total:2002) [2025-01-08 00:50:49.998906] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:49.998927] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4084, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:2332, total:2341) [2025-01-08 00:50:49.998952] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:49.998980] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4085, [delay/us]: prepare:0, schedule:0, submit:600000000, device:2332, total:2341) [2025-01-08 00:50:49.998997] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:49.999012] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:49.999021] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:49.999029] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:49.999038] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=227kB/s, net_out=213kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:50.009936] INFO [SHARE] get_server_tenant_configs (ob_unit_getter.cpp:243) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=8] get_server_tenant_configs(unit_infos=[{unit:unit_id:1, resource_pool_id:1, unit_group_id:1, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, pool:{resource_pool_id:1, name:"sys_pool", unit_count:1, unit_config_id:1, zone_list:["zone1", "zone2", "zone3"], tenant_id:1, replica_type:0}}, {unit:unit_id:1001, resource_pool_id:1001, unit_group_id:1001, zone:"zone1", server:"192.168.159.137:2882", migrate_from_server:"0.0.0.0:0", is_manual_migrate:false, status:0, replica_type:0, config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, pool:{resource_pool_id:1001, name:"pool_1", unit_count:1, unit_config_id:1001, zone_list:["zone1", "zone2", "zone3"], tenant_id:1002, replica_type:0}}]) [2025-01-08 00:50:50.010034] INFO [SHARE] get_compat_mode (ob_unit_getter.cpp:828) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=88] jx_debug: get tenant compatibility mode(tenant_id=1002, compat_mode=0) [2025-01-08 00:50:50.010052] INFO [SHARE] divide_meta_tenant (ob_unit_resource.cpp:1127) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=14] divide meta tenant resource finish(ret=0, ret="OB_SUCCESS", meta_resource={min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, user_resource={min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }, unit_resource={min_cpu:4, max_cpu:4, memory_size:"4GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }) [2025-01-08 00:50:50.010075] INFO [SHARE] divide_meta_tenant (ob_unit_getter.cpp:134) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=21] divide meta tenant finish(ret=0, ret="OB_SUCCESS", meta_tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}, user_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:0, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:50.011739] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28E-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:50.011785] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28E-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326249485990004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326249485990004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:50.014422] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=94] refresh resource plan success(tenant_id=1, plan_name=, directives=[]) [2025-01-08 00:50:50.016327] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=47] refresh resource user mapping rule(tenant_id=1, plan=, user_rules=[]) [2025-01-08 00:50:50.018679] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=43] refresh resource mapping rule(tenant_id=1, plan=, rules=[]) [2025-01-08 00:50:50.018719] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1, plan=) [2025-01-08 00:50:50.020145] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084874-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=30, max_blocked_buffer_size_=9830400, max_buffer_size_=19660800) [2025-01-08 00:50:50.020287] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084874-0-0] [lt=39] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:50:50.020308] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084874-0-0] [lt=18] tenant dfc status(ret=0, get_tenant_id()=1, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=30, get_max_blocked_buffer_size()=9830400, get_max_buffer_size()=19660800, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:50:50.023285] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50521][T1001_Occam][T1001][YB42C0A89F89-00062B2D1D684B89-0-0] [lt=23] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:50.025316] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=13] refresh resource plan success(tenant_id=1001, plan_name=, directives=[]) [2025-01-08 00:50:50.026273] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A84-0-0] [lt=37] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3661) [2025-01-08 00:50:50.027506] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=46] refresh resource user mapping rule(tenant_id=1001, plan=, user_rules=[]) [2025-01-08 00:50:50.029183] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=47] refresh resource mapping rule(tenant_id=1001, plan=, rules=[]) [2025-01-08 00:50:50.029462] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=276] refresh resource mapping rule success(tenant_id=1001, plan=) [2025-01-08 00:50:50.031522] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084874-0-0] [lt=12] trace tenant dfc parameters(max_parallel_cnt_=10, max_blocked_buffer_size_=3276800, max_buffer_size_=6553600) [2025-01-08 00:50:50.031611] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084874-0-0] [lt=38] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:50:50.031626] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084874-0-0] [lt=13] tenant dfc status(ret=0, get_tenant_id()=1001, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=10, get_max_blocked_buffer_size()=3276800, get_max_buffer_size()=6553600, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:50:50.034609] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:42999713}], scn_array=[cnt:1, {val:1736326250001028000, v:0}], curr_size=749, accum_size=7118, time_cost=29045) [2025-01-08 00:50:50.035456] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:84157402}], scn_array=[cnt:1, {val:1736326249904148000, v:0}], curr_size=122, accum_size=1537, time_cost=11811) [2025-01-08 00:50:50.038418] INFO [SHARE] refresh_resource_plan (ob_resource_plan_manager.cpp:176) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=11] refresh resource plan success(tenant_id=1002, plan_name=, directives=[]) [2025-01-08 00:50:50.040276] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=107] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:50.040978] INFO [SHARE] refresh_resource_user_mapping_rule (ob_resource_mapping_rule_manager.cpp:189) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=40] refresh resource user mapping rule(tenant_id=1002, plan=, user_rules=[]) [2025-01-08 00:50:50.042802] INFO [SHARE] refresh_resource_function_mapping_rule (ob_resource_mapping_rule_manager.cpp:227) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=26] refresh resource mapping rule(tenant_id=1002, plan=, rules=[]) [2025-01-08 00:50:50.042842] INFO [SHARE] refresh_resource_mapping_rule (ob_resource_mapping_rule_manager.cpp:149) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=39] refresh resource mapping rule success(tenant_id=1002, plan=) [2025-01-08 00:50:50.044456] INFO [SQL.DTL] calc_max_buffer (ob_dtl_fc_server.cpp:155) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084874-0-0] [lt=10] trace tenant dfc parameters(max_parallel_cnt_=40, max_blocked_buffer_size_=13107200, max_buffer_size_=26214400) [2025-01-08 00:50:50.044546] INFO [SQL.DTL] auto_free_on_time (ob_dtl_tenant_mem_manager.cpp:232) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084874-0-0] [lt=39] auto free to reserve buffer count(reserve_buffer_min_size=16777216, max_reserve_count=2, buffer_size=65536, ret=0) [2025-01-08 00:50:50.044563] INFO [SQL.DTL] clean_on_timeout (ob_dtl_fc_server.cpp:127) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084874-0-0] [lt=15] tenant dfc status(ret=0, get_tenant_id()=1002, get_channel_cnt()=0, get_current_buffer_used()=0, get_current_blocked_cnt()=0, get_current_buffer_cnt()=0, get_max_parallel()=40, get_max_blocked_buffer_size()=13107200, get_max_buffer_size()=26214400, get_accumulated_blocked_cnt()=0, get_max_size_per_channel()=655360) [2025-01-08 00:50:50.044585] INFO [SERVER.OMT] periodically_check_tenant (ob_tenant_node_balancer.cpp:529) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=12] refresh resource manager plan(plan_mgr=background_quota:2147483647, tenant_plan_map:{1->{value:""}, 1001->{value:""}, 1002->{value:""}}, rule_mgr=user_rule_map:{} function_rule_map:{} group_id_name_map:{} group_name_id_map:{}, col_rule_mgr={{v:1}->tenant_id:1, version: 0, rule_id_map:{} group_id_map:{} {v:1001}->tenant_id:1001, version: 0, rule_id_map:{} group_id_map:{} {v:1002}->tenant_id:1002, version: 0, rule_id_map:{} group_id_map:{} }) [2025-01-08 00:50:50.045289] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:112) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=700] refresh tenant units(sys_unit_cnt=3, units=[{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}, {tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}], ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:50.045344] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=45] finish set nway(tenant_id_=1, nway=3) [2025-01-08 00:50:50.045355] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=9] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1, nway=3, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1736322513705891, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:50.045376] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084874-0-0] [lt=17] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=1.638399963378906250e+02, this=0x2ad8955e6170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=819, Resource Limit(MB)=819, Throttle Trigger(MB)=491, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=3.948457122979588263e+01) [2025-01-08 00:50:50.045395] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1][YB42C0A89F89-00062B2D16084874-0-0] [lt=19] succ to set_memstore_threshold of memstore allocator(tenant_id=1, ret=0) [2025-01-08 00:50:50.045406] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=7] finish set nway(tenant_id_=1002, nway=4) [2025-01-08 00:50:50.045412] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=6] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1002, nway=4, new_tma_limit=1932735240, pre_tma_limit=1932735240, cur_memstore_limit_percent=40, tenant_config={tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:4, max_cpu:4, memory_size:"3GB", log_disk_size:"4.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:50.045428] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084874-0-0] [lt=14] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=2.457599945068359375e+02, this=0x2ad8f309c170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=1228, Resource Limit(MB)=1228, Throttle Trigger(MB)=737, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.831060728156441719e+00) [2025-01-08 00:50:50.045439] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1002][YB42C0A89F89-00062B2D16084874-0-0] [lt=10] succ to set_memstore_threshold of memstore allocator(tenant_id=1002, ret=0) [2025-01-08 00:50:50.045447] INFO set_nway (ob_tenant_mutil_allocator.cpp:392) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=4] finish set nway(tenant_id_=1001, nway=1) [2025-01-08 00:50:50.045452] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:403) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=5] ObTenantMutilAllocator update tenant mem_limit finished(ret=0, tenant_id=1001, nway=1, new_tma_limit=644245080, pre_tma_limit=644245080, cur_memstore_limit_percent=40, tenant_config={tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"unit_1", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"0.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:4, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:4, }}, mode:0, create_timestamp:1736324329143620, is_removed:false, hidden_sys_data_disk_config_size:0}) [2025-01-08 00:50:50.045466] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:287) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084874-0-0] [lt=13] [Throttle] Update Throttle Unit Config(is_adaptive_update=false, N=8.191999816894531250e+01, this=0x2ad8fb762170, enable_adaptive_limit_=false, Unit Name=Memstore, Config Specify Resource Limit(MB)=409, Resource Limit(MB)=409, Throttle Trigger(MB)=245, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=6.241572899671315326e+02) [2025-01-08 00:50:50.045477] INFO update_tenant_mem_limit (ob_tenant_mutil_allocator_mgr.cpp:412) [46263][OmtNodeBalancer][T1001][YB42C0A89F89-00062B2D16084874-0-0] [lt=11] succ to set_memstore_threshold of memstore allocator(tenant_id=1001, ret=0) [2025-01-08 00:50:50.046759] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=12] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:50:50.046799] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=37] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1001, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:50:50.046811] INFO [SERVER.OMT] refresh_tenants (ob_tenant_config_mgr.cpp:225) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=10] [DATA_VERSION] periodically update data_version(tmp_ret=0, tmp_ret="OB_SUCCESS", tenant_id=1002, value_updated=true, data_version={version_str:"4.3.5.0", version_val:17180067072}) [2025-01-08 00:50:50.046847] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=8] cal_all_part_disk_default_percentage succ(data_dir="/ob/data/sstable", clog_dir="/ob/data/clog", shared_mode=true, data_disk_total_size=34960134144, data_disk_default_percentage=60, clog_disk_total_size=34960134144, clog_disk_default_percentage=30) [2025-01-08 00:50:50.046860] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=12] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=34960134144, disk_size=15032385536) [2025-01-08 00:50:50.046869] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=8] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=34960134144) [2025-01-08 00:50:50.046881] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:800) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=8] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=34960134144, this={dir::"/ob/data/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:13, max_block_id:224, min_log_disk_size_for_all_tenants_:7516192768, is_inited:true}) [2025-01-08 00:50:50.046896] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:136) [46263][OmtNodeBalancer][T0][YB42C0A89F89-00062B2D16084874-0-0] [lt=14] refresh tenant config(tenants=[1, 1001, 1002], ret=0) [2025-01-08 00:50:50.047143] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=12, avg_log_batch_cnt=1, total_group_log_size=1877, avg_group_log_size=156) [2025-01-08 00:50:50.047278] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=20] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1877) [2025-01-08 00:50:50.048846] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=185] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1984, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:50.048982] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=129] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=611, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:50.049022] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=22] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=11, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:50.049054] INFO [STORAGE.REDO] write_logs (ob_storage_log_writer.cpp:512) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=41] Slow write(duration=10471, write_len=4096) [2025-01-08 00:50:50.049069] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=14] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8551, offset=2425397}, end_cursor:ObLogCursor{file_id=1, log_id=8552, offset=2425694}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89468c050, len:3531, log_data_len:297, seq:8551, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:50.051906] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=16] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8552, offset=2425694}, end_cursor:ObLogCursor{file_id=1, log_id=8553, offset=2425991}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894690050, len:3234, log_data_len:297, seq:8552, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:50.052298] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36B7-0-0] [lt=39] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1342) [2025-01-08 00:50:50.052551] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17687, miss_cnt:0, cache_read_size:2163733, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:50.052913] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=83] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1454, hit_cnt=12, read_cnt=12, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.055819] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=46] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:50.063862] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-01-08 00:50:50.080407] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=90] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:50.096965] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.130799] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6752137216, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:50.147046] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:50.153520] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36B9-0-0] [lt=24] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=17, avg_log_submit_to_flush_time=3512, avg_log_submit_to_slide_time=56682) [2025-01-08 00:50:50.153563] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58329][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36B9-0-0] [lt=41][errcode=-4389] log_task life cost too much time(palf_id=1003, self="192.168.159.137:2882", log_id=17777, log_task={header:{begin_lsn:{lsn:2163733}, end_lsn:{lsn:2163855}, log_id:17777, min_scn:{val:1736326249699974003, v:0}, max_scn:{val:1736326249699974003, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2163611}, prev_proposal_id:1, committed_end_lsn:{lsn:2163733}, data_checksum:940009244, accum_checksum:2879697556, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326250052281, freeze_ts:1736326250052281, submit_ts:1736326250052288, flushed_ts:1736326250056122, gen_to_freeze cost time:0, gen_to_submit cost time:7, submit_to_flush cost time:3834}, fs_cb_begin_ts=1736326250153510, log_life_time=101229) [2025-01-08 00:50:50.153684] INFO [PALF] read (log_cache.cpp:103) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=152] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.157699] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50460][T1002_TenantWea][T1001][YB42C0A89F89-00062B2D2068ABDE-0-0] [lt=78] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.766940498352050781e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=388, Throttle Trigger(MB)=233, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.714020632270641045e+02) [2025-01-08 00:50:50.158254] INFO [LIB] stat (utility.h:1210) [50460][T1002_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:50.161839] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:3652K/907:229920 disk:3652K/907:229920 t1m2:3368K/841:205468 t1m2g0:3368K/841:205468 t1002m2:196K/47:439630 t1002m2g0:196K/47:439630 t1001m2:88K/19:768426 t1001m2g0:88K/19:768426 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:50.162662] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58339][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:50.162712] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2025-01-08 00:50:50.164475] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get_max_applied_scn(scn={val:1736326249904148001, v:0}, this={ls_id_:{id:1001}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:2141388}, palf_committed_end_scn_:{val:1736326249904148002, v:0}, last_check_scn_:{val:1736326249904148002, v:0}, max_applied_cb_scn_:{val:1736326249904148001, v:0}}) [2025-01-08 00:50:50.164514] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1736324436756804000, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2141388}, readable_log_tail:{lsn:2141388}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64963380, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:50.164671] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread created(tenant_->id()=1001, group_id_=1, token=5) [2025-01-08 00:50:50.170194] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC368-0-0] [lt=50][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.172385] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43F7-0-0] [lt=20] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=36, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:50.172425] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43F7-0-0] [lt=38] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=12954, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:50.172438] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43F7-0-0] [lt=11] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=36, avg_log_submit_to_flush_time=2147, avg_log_submit_to_slide_time=12918) [2025-01-08 00:50:50.172550] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1211, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.175943] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A46-0-0] [lt=102] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.187324] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46478][T1_L0_G5][T1][YB42C0A89F8B-00062B2D160BB023-0-0] [lt=86] get number(ret=0, service_type_=0, range=1, base_id=1736326250187003000, start_id=1736326250187003000, end_id=1736326250187003001) [2025-01-08 00:50:50.188853] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=28] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1001, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:50.190346] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=99] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:50.190462] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=41] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326244258401000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:50.190518] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=54] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326244258401000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1001, freeze_info:{latest_snapshot_gc_scn:{val:1736326244258401000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:50:50.196573] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50691][T1001_L0_G5][T1001][YB42C0A89F8A-00062B2D1609BCBB-0-0] [lt=146] get number(ret=0, service_type_=0, range=1, base_id=1736326250196395000, start_id=1736326250196395000, end_id=1736326250196395001) [2025-01-08 00:50:50.197245] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.198405] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58340][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:50.199463] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] worker thread created(tenant_->id()=1001, group_id_=1, token=6) [2025-01-08 00:50:50.203530] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50913][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE7C9-0-0] [lt=35] get number(ret=0, service_type_=0, range=1, base_id=1736326250202295000, start_id=1736326250202295000, end_id=1736326250202295001) [2025-01-08 00:50:50.211619] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A48-0-0] [lt=23] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.214782] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=80] gc stale ls task succ [2025-01-08 00:50:50.221130] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:50.221182] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:50.221798] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB44-0-0] [lt=24][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193755135630, cache_obj->added_lc()=false, cache_obj->get_object_id()=13551, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:50.224889] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A49-0-0] [lt=45] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.227145] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A4A-0-0] [lt=82] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.228684] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=3661, avg_group_log_size=332) [2025-01-08 00:50:50.228770] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=81] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=3661) [2025-01-08 00:50:50.228790] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=15] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:50.230511] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB8700D-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.230575] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:50.230589] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:50.236542] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A4B-0-0] [lt=16] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.242468] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1989) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=29] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-01-08 00:50:50.242511] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1491) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=38] start lib cache evict(tenant_id=1, mem_hold=93765632, mem_limit=107374180, cache_obj_num=300, cache_node_num=300) [2025-01-08 00:50:50.242525] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1508) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=93765632, mem_limit=107374180, cache_obj_num=300, cache_node_num=300) [2025-01-08 00:50:50.242537] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2775) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:50.245449] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=45] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:50.245453] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2493) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=7] Dumping All Cache Objs(alloc_obj_list.count()=300, alloc_obj_list=[{obj_id:2842, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180042}, {obj_id:2927, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179618}, {obj_id:5613, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202676}, {obj_id:5935, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194846}, {obj_id:6719, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192882}, {obj_id:6728, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6730, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:6733, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:7580, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190304}, {obj_id:7809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201848}, {obj_id:7812, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195439}, {obj_id:7873, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179416}, {obj_id:7882, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194188}, {obj_id:7884, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179156}, {obj_id:7886, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186967}, {obj_id:7906, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189321}, {obj_id:7907, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187042}, {obj_id:7910, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178466}, {obj_id:7915, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206569}, {obj_id:7934, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:160477}, {obj_id:7941, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224899}, {obj_id:7942, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271983}, {obj_id:7943, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186069}, {obj_id:7948, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212298}, {obj_id:7954, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190678}, {obj_id:7984, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203552}, {obj_id:7985, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186878}, {obj_id:7989, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180152}, {obj_id:7992, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182846}, {obj_id:7995, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177811}, {obj_id:7997, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177804}, {obj_id:7998, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177848}, {obj_id:7999, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177807}, {obj_id:8000, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178930}, {obj_id:8005, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181004}, {obj_id:8006, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177577}, {obj_id:8009, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179162}, {obj_id:8017, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176432}, {obj_id:8018, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176080}, {obj_id:8019, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:210477}, {obj_id:8028, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175312}, {obj_id:8064, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191402}, {obj_id:8240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184955}, {obj_id:8242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179248}, {obj_id:8266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177580}, {obj_id:8271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177672}, {obj_id:8278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:8286, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:208698}, {obj_id:8287, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:211220}, {obj_id:8294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184352}, {obj_id:8311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185357}, {obj_id:8312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183713}, {obj_id:8325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283032}, {obj_id:8333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179968}, {obj_id:8335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183610}, {obj_id:8339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233859}, {obj_id:8347, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:235206}, {obj_id:8372, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178632}, {obj_id:9060, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191961}, {obj_id:9627, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200513}, {obj_id:9640, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190450}, {obj_id:11987, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176763}, {obj_id:11988, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13022, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190209}, {obj_id:13168, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13169, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:161246}, {obj_id:13173, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188140}, {obj_id:13177, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191323}, {obj_id:13178, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13180, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195299}, {obj_id:13184, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188613}, {obj_id:13185, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:570010}, {obj_id:13191, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:11244645}, {obj_id:13196, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177352}, {obj_id:13205, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:411582}, {obj_id:13206, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177608}, {obj_id:13208, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189960}, {obj_id:13209, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200592}, {obj_id:13210, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199510}, {obj_id:13211, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13212, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185015}, {obj_id:13213, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:656293}, {obj_id:13214, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196013}, {obj_id:13215, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13216, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190234}, {obj_id:13217, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193397}, {obj_id:13218, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13219, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655888}, {obj_id:13220, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:655966}, {obj_id:13221, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13222, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:283956}, {obj_id:13223, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187914}, {obj_id:13224, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178744}, {obj_id:13225, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183047}, {obj_id:13226, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:350952}, {obj_id:13227, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212956}, {obj_id:13228, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202039}, {obj_id:13229, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200992}, {obj_id:13230, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185681}, {obj_id:13231, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191932}, {obj_id:13232, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177152}, {obj_id:13233, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177800}, {obj_id:13234, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193355}, {obj_id:13235, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183584}, {obj_id:13236, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181979}, {obj_id:13237, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:201844}, {obj_id:13238, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:272008}, {obj_id:13239, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182200}, {obj_id:13240, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182642}, {obj_id:13241, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180624}, {obj_id:13242, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177952}, {obj_id:13243, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198280}, {obj_id:13244, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182506}, {obj_id:13245, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177272}, {obj_id:13246, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175888}, {obj_id:13247, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185486}, {obj_id:13248, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182545}, {obj_id:13249, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177104}, {obj_id:13250, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13251, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176176}, {obj_id:13252, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13254, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182749}, {obj_id:13255, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182518}, {obj_id:13256, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13257, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182530}, {obj_id:13258, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13259, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182580}, {obj_id:13260, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182456}, {obj_id:13261, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182789}, {obj_id:13262, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182858}, {obj_id:13263, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182542}, {obj_id:13264, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182875}, {obj_id:13265, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182526}, {obj_id:13266, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182569}, {obj_id:13267, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182514}, {obj_id:13268, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13269, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182500}, {obj_id:13270, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176752}, {obj_id:13271, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182512}, {obj_id:13272, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182503}, {obj_id:13273, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13274, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182561}, {obj_id:13275, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182567}, {obj_id:13276, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13277, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194665}, {obj_id:13278, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182482}, {obj_id:13279, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179136}, {obj_id:13280, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177462}, {obj_id:13281, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177200}, {obj_id:13282, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13283, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182952}, {obj_id:13284, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182713}, {obj_id:13285, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248328}, {obj_id:13288, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182595}, {obj_id:13289, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13290, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:226000}, {obj_id:13291, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182566}, {obj_id:13292, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182537}, {obj_id:13293, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182496}, {obj_id:13294, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182528}, {obj_id:13295, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183665}, {obj_id:13296, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182498}, {obj_id:13297, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186224}, {obj_id:13298, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182513}, {obj_id:13299, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194020}, {obj_id:13300, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182507}, {obj_id:13301, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181283}, {obj_id:13302, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182587}, {obj_id:13303, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182505}, {obj_id:13304, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182473}, {obj_id:13305, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182491}, {obj_id:13307, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174944}, {obj_id:13308, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183034}, {obj_id:13309, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182511}, {obj_id:13310, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182575}, {obj_id:13311, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182848}, {obj_id:13312, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182560}, {obj_id:13313, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182919}, {obj_id:13314, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182835}, {obj_id:13315, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182637}, {obj_id:13316, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182531}, {obj_id:13317, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13318, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182558}, {obj_id:13319, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182602}, {obj_id:13321, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182510}, {obj_id:13322, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182821}, {obj_id:13323, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182833}, {obj_id:13324, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183391}, {obj_id:13325, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182998}, {obj_id:13326, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182769}, {obj_id:13327, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182606}, {obj_id:13328, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182633}, {obj_id:13329, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183069}, {obj_id:13330, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182638}, {obj_id:13331, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183466}, {obj_id:13332, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13333, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182476}, {obj_id:13334, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:289839}, {obj_id:13335, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:636225}, {obj_id:13336, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:412548}, {obj_id:13337, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198834}, {obj_id:13338, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176448}, {obj_id:13339, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194581}, {obj_id:13341, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183470}, {obj_id:13342, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213471}, {obj_id:13343, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177590}, {obj_id:13344, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177627}, {obj_id:13345, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177992}, {obj_id:13346, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192568}, {obj_id:13349, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185378}, {obj_id:13350, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203417}, {obj_id:13351, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185584}, {obj_id:13352, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178857}, {obj_id:13353, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185263}, {obj_id:13354, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180782}, {obj_id:13355, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188428}, {obj_id:13356, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193448}, {obj_id:13357, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13358, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178272}, {obj_id:13359, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13360, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182249}, {obj_id:13361, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248168}, {obj_id:13362, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13364, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179211}, {obj_id:13365, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179898}, {obj_id:13369, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182768}, {obj_id:13370, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187536}, {obj_id:13371, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:225221}, {obj_id:13375, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203415}, {obj_id:13376, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196906}, {obj_id:13378, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193243}, {obj_id:13379, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189042}, {obj_id:13382, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:217816}, {obj_id:13388, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180539}, {obj_id:13389, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180070}, {obj_id:13390, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188983}, {obj_id:13391, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183990}, {obj_id:13392, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185221}, {obj_id:13393, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193108}, {obj_id:13394, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245604}, {obj_id:13395, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:200660}, {obj_id:13396, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:284537}, {obj_id:13397, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:279835}, {obj_id:13398, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184493}, {obj_id:13399, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188856}, {obj_id:13400, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199144}, {obj_id:13401, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:248228}, {obj_id:13402, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178291}, {obj_id:13403, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:317177}, {obj_id:13404, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13405, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175776}, {obj_id:13406, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:323228}, {obj_id:13407, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13408, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:295843}, {obj_id:13409, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179606}, {obj_id:13410, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178515}, {obj_id:13411, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180292}, {obj_id:13412, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13414, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13415, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183657}, {obj_id:13416, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199853}, {obj_id:13417, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187468}, {obj_id:13421, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179456}, {obj_id:13422, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183529}, {obj_id:13423, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274798}, {obj_id:13426, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177088}, {obj_id:13432, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:215055}, {obj_id:13433, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206005}, {obj_id:13434, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:177136}, {obj_id:13435, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184151}, {obj_id:13436, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184118}, {obj_id:13438, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189857}, {obj_id:13446, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181408}, {obj_id:13447, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189124}, {obj_id:13449, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13450, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13452, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13453, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13455, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13456, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13457, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13458, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13460, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193917}, {obj_id:13461, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199156}, {obj_id:13481, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:206237}, {obj_id:13482, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184745}, {obj_id:13493, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184214}, {obj_id:13494, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180450}, {obj_id:13504, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195868}, {obj_id:13505, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191698}, {obj_id:13509, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1752}, {obj_id:13524, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224140}, {obj_id:13525, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183490}, {obj_id:13527, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:203816}, {obj_id:13532, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180203}, {obj_id:13533, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182096}, {obj_id:13534, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182624}, {obj_id:13535, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176208}, {obj_id:13536, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204027}]) [2025-01-08 00:50:50.246297] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2783) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=797] schedule next cache evict task(evict_interval=5000000) [2025-01-08 00:50:50.246636] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=29] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:50.246731] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=19] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326245675194000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:50.247161] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50916][T1002_L0_G5][T1002][YB42C0A89F8A-00062B2D1609BCBB-0-0] [lt=43] get number(ret=0, service_type_=0, range=1, base_id=1736326250241481000, start_id=1736326250241481000, end_id=1736326250241481001) [2025-01-08 00:50:50.262385] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=33] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:50.279631] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C98-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:50.279670] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C98-0-0] [lt=36] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:50.279707] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=13] gc stale ls task succ [2025-01-08 00:50:50.291974] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540D-0-0] [lt=16][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.298382] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.300066] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A4C-0-0] [lt=102] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.312555] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A4D-0-0] [lt=46] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.319420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.319463] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.329973] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=36] current monitor number(seq_=-1) [2025-01-08 00:50:50.331314] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:50.331716] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A4E-0-0] [lt=95] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.333115] INFO [SERVER] get_dbms_sched_job_infos_in_tenant (ob_dbms_sched_table_operator.cpp:682) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=24] failed to get result(ret=-4008) [2025-01-08 00:50:50.333408] INFO [SERVER] check_new_jobs (ob_dbms_sched_job_master.cpp:528) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=16] check new jobs(ret=0, tenant_id=1001, is_oracle_tenant=false, job_infos=[{tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:9, job_name_:"ASYNC_GATHER_STATS_JOB_PROC", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736326185605925, start_date_:1736325283345722, last_date_:1736326183353587, this_date_:0, next_date_:1736327083345722, end_date_:64060560000000000, total_:6596354, interval_:"FREQ=MINUTELY; INTERVAL=15", repeat_interval_:"FREQ=MINUTELY; INTERVAL=15", failures_:0, flag_:0, what_:"DBMS_STATS.ASYNC_GATHER_STATS_JOB_PROC(600000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:900000000, max_failures_:0, state_:"SCHEDULED"}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:5, job_name_:"FRIDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736517600000000, last_date_:0, this_date_:0, next_date_:1736517600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:1, job_name_:"MONDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736776800000000, last_date_:0, this_date_:0, next_date_:1736776800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:8, job_name_:"OPT_STATS_HISTORY_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736410783345628, last_date_:0, this_date_:0, next_date_:1736410783345628, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.PURGE_STATS(NULL)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:43200, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:6, job_name_:"SATURDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736546400000000, last_date_:0, this_date_:0, next_date_:1736546400000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:10, job_name_:"SPM_STATS_MANAGER", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736355600000000, last_date_:0, this_date_:0, next_date_:1736355600000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=DAYLY; INTERVAL=1", repeat_interval_:"FREQ=DAYLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_SPM.HANDLE_SPM_STATS_JOB_PROC()", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:600, interval_ts_:86400000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:7, job_name_:"SUNDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736632800000000, last_date_:0, this_date_:0, next_date_:1736632800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(72000000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:72000, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:4, job_name_:"THURSDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736431200000000, last_date_:0, this_date_:0, next_date_:1736431200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:2, job_name_:"TUESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736863200000000, last_date_:0, this_date_:0, next_date_:1736863200000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}, {tenant_id_:1001, user_id_:18446744073709551615, database_id_:18446744073709551615, job_:3, job_name_:"WEDNESDAY_WINDOW", job_style_:"REGULAR", job_type_:"", lowner_:"root@%", powner_:"root@%", cowner_:"oceanbase", last_modify_:1736324383356410, start_date_:1736344800000000, last_date_:0, this_date_:0, next_date_:1736344800000000, end_date_:64060560000000000, total_:0, interval_:"FREQ=WEEKLY; INTERVAL=1", repeat_interval_:"FREQ=WEEKLY; INTERVAL=1", failures_:0, flag_:0, what_:"DBMS_STATS.GATHER_DATABASE_STATS_JOB_PROC(14400000000)", nlsenv_:"", charenv_:"", field1_:"", scheduler_flags_:0, enabled_:true, auto_drop_:false, max_run_duration_:14400, interval_ts_:604800000000, max_failures_:0, state_:""}]) [2025-01-08 00:50:50.333690] INFO [SERVER] check_tenant (ob_dbms_sched_job_master.cpp:517) [50607][T1001_DBMSSched][T1001][Y0-0000000000000000-0-0] [lt=278] check all tenants(ret=0) [2025-01-08 00:50:50.339539] INFO [SERVER] scheduler (ob_dbms_sched_job_master.cpp:219) [50607][T1001_DBMSSched][T1001][YB42C0A89F89-00062B2D1F784A50-0-0] [lt=9] success to scheduler single dbms sched job(ret=0, tmp_ret=0) [2025-01-08 00:50:50.341507] INFO [SHARE] get_freeze_info_larger_or_equal_than (ob_freeze_info_proxy.cpp:168) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=66] finish load_freeze_info(ret=0, ret="OB_SUCCESS", tenant_id=1002, sql=SELECT * FROM __all_freeze_info WHERE frozen_scn >= 1 ORDER BY frozen_scn) [2025-01-08 00:50:50.342681] INFO [SHARE] get_all_snapshots (ob_snapshot_table_proxy.cpp:428) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=49] get all snapshots(ret=0, snapshots.count()=0, snapshots=[]) [2025-01-08 00:50:50.342793] INFO [SHARE] update_freeze_info (ob_freeze_info_manager.cpp:219) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=39] inner load succ(latest_freeze_info={frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}, freeze_info_={latest_snapshot_gc_scn:{val:1736326245564663000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}) [2025-01-08 00:50:50.342821] INFO [STORAGE] inner_update_info (ob_tenant_freeze_info_mgr.cpp:719) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=25] ObTenantFreezeInfoMgr success to update infos(new_snapshot_gc_scn={val:1736326245564663000, v:0}, new_freeze_infos=[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}], new_snapshots=[], freeze_info_mgr_={is_inited:true, tenant_id:1002, freeze_info:{latest_snapshot_gc_scn:{val:1736326245564663000, v:0}, frozen_statuses_.count():1, frozen_statuses:[{frozen_version:{val:1, v:0}, schema_version:1, data_version:17180067072}]}}) [2025-01-08 00:50:50.343136] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD84-0-0] [lt=28] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326250435371, keepalive_ts:-1}) [2025-01-08 00:50:50.343162] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D1809BD84-0-0] [lt=24] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326250435371, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:50.347682] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=77, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:50.353782] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=19] gc stale ls task succ [2025-01-08 00:50:50.365662] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:18639483}], scn_array=[cnt:1, {val:1736326249904148004, v:0}], curr_size=122, accum_size=610, time_cost=11047) [2025-01-08 00:50:50.368137] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2228) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, lsn_array=[cnt:1, {lsn:2164099}], scn_array=[cnt:1, {val:1736326250308616000, v:0}], curr_size=122, accum_size=610, time_cost=13288) [2025-01-08 00:50:50.368275] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2025-01-08 00:50:50.369226] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=40] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4120, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:50.369255] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=27] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=9757) [2025-01-08 00:50:50.369288] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=13] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=542, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:50.369368] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=23] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=24, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:50.371831] INFO [LIB] stat (utility.h:1210) [58241][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37E7-0-0] [lt=11] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=11, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:50.371855] INFO [LIB] stat (utility.h:1210) [58241][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37E7-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=17, stat_interval=1000000, avg (count/cost)=6078, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:50.371865] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58241][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37E7-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=12, avg_log_gen_to_submit_time=118, avg_log_submit_to_flush_time=4864, avg_log_submit_to_slide_time=5832) [2025-01-08 00:50:50.373008] INFO [LIB] stat (utility.h:1210) [58245][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1B5BBA44-0-0] [lt=30] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:50.373280] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E4-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:50.373321] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E4-0-0] [lt=24] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326250308615999, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326250308615999, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:50.376522] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=9757, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.376891] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=76] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326249699974003, v:0}, new_max_version={val:1736326249699974003, v:0}, last_min_version={val:1736326249404190004, v:0}, last_max_version={val:1736326249404190004, v:0}, total_time_us=11072, delta_us=676914) [2025-01-08 00:50:50.376921] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326249699974003, v:0}, min_version={val:1736326249699974003, v:0}, max_version={val:1736326249699974003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:50.381372] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27] worker thread created(tenant_->id()=1001, group_id_=1, token=7) [2025-01-08 00:50:50.381375] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58341][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:50.386163] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [46337][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=49] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad89c1e22b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:50:50.389948] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=27] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:50.389988] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-01-08 00:50:50.390000] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:50.390015] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:50.390030] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193754303479, flush_io_finished_ret_:0, flush_io_finished_round_:1909, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:50.391090] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=9] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:50.391140] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=37] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:50.391182] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=16] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=109) [2025-01-08 00:50:50.393024] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37E8-0-0] [lt=23] init point thread id with(&point=0x5572d50e2280, idx_=997, point=[thread id=58341, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58341) [2025-01-08 00:50:50.394925] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:50.394958] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-01-08 00:50:50.394968] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:50.394982] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:50.394997] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193754307012, flush_io_finished_ret_:0, flush_io_finished_round_:3737, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:50.395108] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=6] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:50.398589] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.403925] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=176] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:50.404027] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=88] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:50.404043] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=14] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-01-08 00:50:50.404050] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-01-08 00:50:50.404111] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=5] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-01-08 00:50:50.404130] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=17] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=94) [2025-01-08 00:50:50.407162] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=132] gts statistics(tenant_id=1, gts_rpc_cnt=10, get_gts_cache_cnt=503, get_gts_with_stc_cnt=32, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=30, try_wait_gts_elapse_cnt=0) [2025-01-08 00:50:50.407366] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=133] gts statistics(tenant_id=1001, gts_rpc_cnt=10, get_gts_cache_cnt=532, get_gts_with_stc_cnt=73, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=73, try_wait_gts_elapse_cnt=0) [2025-01-08 00:50:50.433458] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58342][][T1][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:50.433611] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38] worker thread created(tenant_->id()=1, group_id_=1, token=2) [2025-01-08 00:50:50.452257] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:50.452783] INFO [COMMON] try_expand_thread_count (ob_dynamic_thread_pool.cpp:473) [46231][][T0][Y0-0000000000000000-0-0] [lt=20] expand thread count(this=0x2ad8885abcf0, max_thread_cnt=96, cur_thread_count=2, inc_cnt=1, queue_size=2) [2025-01-08 00:50:50.452841] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] submit pre barrier log success(log_type=17, ls_id={id:1003}, lsn={lsn:2164277}, scn={val:1736326250308616001, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:50:50.452915] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:984) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=51] wait for barrier(ret=-4023, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:16, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326250452692, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2164333}, next_to_submit_scn_:{val:1736326250308616002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2164221}, end_lsn:{lsn:2164333}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2164333}, readable_log_tail:{lsn:2164333}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64940435, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326250308616001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326250308616001, v:0}, data_checksum:2012734446, flag:54327}}, init_mode_version:0, accumulate_checksum:2912715304, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326250308616001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17692, miss_cnt:0, cache_read_size:2164333, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2164277}, scn_:{val:1736326250308616001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326250364376, first_handle_ts_:1736326250452913, replay_cost_:-1, retry_cost_:-1, read_log_buf_:0x2ad94da04150, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}) [2025-01-08 00:50:50.453223] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] success to update reserved snapshot(ret=0, ls_id={id:1003}, min_reserved_snapshot_=1736324445564663000) [2025-01-08 00:50:50.453299] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50483][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=73] pre barrier log replay succ(replay_task={ls_id_:{id:1003}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2164277}, scn_:{val:1736326250308616001, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326250364376, first_handle_ts_:1736326250453206, replay_cost_:90, retry_cost_:-1, read_log_buf_:0x2ad94da04160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326250452692, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2164333}, next_to_submit_scn_:{val:1736326250308616002, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2164221}, end_lsn:{lsn:2164333}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2164333}, readable_log_tail:{lsn:2164333}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64940435, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326250308616001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326250308616001, v:0}, data_checksum:2012734446, flag:54327}}, init_mode_version:0, accumulate_checksum:2912715304, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326250308616001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17692, miss_cnt:0, cache_read_size:2164333, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=24, ret=0) [2025-01-08 00:50:50.454684] INFO [LIB] do_thread_recycle (threads.cpp:163) [46231][][T0][Y0-0000000000000000-0-0] [lt=28] recycle one thread(this=0x2ad8885abcf0, total=3, remain=2) [2025-01-08 00:50:50.455444] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:50.458352] INFO [CLOG] is_in_sync (ob_log_handler.cpp:709) [46374][T1_L5_G0][T1][YB42C0A89F8A-00062B2D16C9BEC1-0-0] [lt=28] is_in_sync(ret=0, id=1, is_log_sync=true, is_need_rebuild=false) [2025-01-08 00:50:50.463176] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.463263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.464793] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8305-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:50.465323] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1ECC8305-0-0] [lt=527] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=7592, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:50.465688] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.467454] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58342][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A89-0-0] [lt=128] init point thread id with(&point=0x5572d50e22c0, idx_=998, point=[thread id=58342, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58342) [2025-01-08 00:50:50.468596] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F4F-0-0] [lt=33] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:50.468620] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F4F-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=46249, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:50.474856] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43FB-0-0] [lt=41] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:50.491181] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1736326250491177, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007648, add_bytes: 0 [2025-01-08 00:50:50.491668] INFO [BALANCE.TRANSFER] do_work (ob_tenant_transfer_service.cpp:146) [50411][T1002_TntTransf][T1002][YB42C0A89F89-00062B2D22484647-0-0] [lt=36] [TENANT_TRANSFER] finish one round(ret=0, ret="OB_SUCCESS", all_tasks_count=0, thread_task_count=0, thread_idx=0) [2025-01-08 00:50:50.498724] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.520007] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.531708] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:50.533406] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=37] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=9757, avg_group_log_size=542) [2025-01-08 00:50:50.533438] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=26] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=9757) [2025-01-08 00:50:50.536313] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58344][][T1001][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:50.536379] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20] worker thread created(tenant_->id()=1001, group_id_=1, token=8) [2025-01-08 00:50:50.542356] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=253] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.544646] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=39] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.548283] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=49, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:50.548315] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:50.548518] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=198] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:50.548559] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:50.550622] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=16] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:50.551424] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F51-0-0] [lt=46] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=9, avg_log_gen_to_submit_time=112, avg_log_submit_to_flush_time=5820, avg_log_submit_to_slide_time=7442) [2025-01-08 00:50:50.552142] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F52-0-0] [lt=48] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=23, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:50.552241] INFO [LIB] stat (utility.h:1210) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F52-0-0] [lt=97] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=6361, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:50.553306] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36C1-0-0] [lt=24] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=12, avg_log_batch_cnt=0, total_group_log_size=1454, avg_group_log_size=121) [2025-01-08 00:50:50.556058] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:50.562004] INFO [SHARE] force_set_master_rs (ob_rs_mgr.cpp:277) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCD-0-0] [lt=25] [RS_MGR] force set rs list(master_rs="192.168.159.137:2882") [2025-01-08 00:50:50.562052] INFO [SERVER] handle_heartbeat (ob_service.cpp:3753) [46642][T1_L0_G25][T1][YB42C0A89F89-00062B2D18E84CCD-0-0] [lt=40] handle_heartbeat(ret=0, ret="OB_SUCCESS", hb_request={server:"192.168.159.137:2882", server_id:1, rs_addr:"192.168.159.137:2882", rs_server_status:1, epoch_id:1}, hb_response={zone:"zone1", server:"192.168.159.137:2882", sql_port:2881, build_version:"4.3.5.0_100000202024123117-5d6cb5cbc3f7c1ab6eb22e40abec8e160a8764d5(Dec 31 2024 17:35:01)", start_service_time:1736322580423131, server_health_status:{data_disk_status_:1, data_disk_status:"NORMAL"}}, time_cost=52) [2025-01-08 00:50:50.569140] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2025-01-08 00:50:50.569346] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:50.571453] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58344][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CCC2B53-0-0] [lt=38] init point thread id with(&point=0x5572d50e2340, idx_=1000, point=[thread id=58344, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58344) [2025-01-08 00:50:50.575795] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:470) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=91] skip defragment task(ret=0, need_defragment=false) [2025-01-08 00:50:50.575823] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:512) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=26] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=1) [2025-01-08 00:50:50.575834] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:530) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=10] successfully defragment data blocks(rewrite_cnt=0) [2025-01-08 00:50:50.575844] INFO [STORAGE] ~ObITabletFilterOp (ob_tablet_pointer.cpp:765) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=7] tablet skip filter destructed(total_cnt=0, skip_cnt=0, not_in_mem_cnt=0, invalid_attr_cnt=0, total_tablet_cnt=0, total_skip_cnt=0, not_in_mem_tablet_cnt=0, invalid_attr_tablet_cnt=0) [2025-01-08 00:50:50.580607] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=50] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:50.598880] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.616404] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B8B-0-0] [lt=34] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:50.616566] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [50373][T1002_Occam][T1002][YB42C0A89F89-00062B2D1D284B8D-0-0] [lt=37] for each ls(succ_num=4, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:50.631534] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=34] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3242, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:50.631593] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=56] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3661) [2025-01-08 00:50:50.631620] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=250, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:50.631654] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=16] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=30, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:50.632415] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A8A-0-0] [lt=12] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=17, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:50.632442] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A8A-0-0] [lt=26] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=4978, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:50.632453] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58332][T1_L0_G1][T1][YB42C0A89F8B-00062B2D181C5A8A-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=13, avg_log_gen_to_submit_time=150, avg_log_submit_to_flush_time=3581, avg_log_submit_to_slide_time=4827) [2025-01-08 00:50:50.632558] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3661, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:50.647289] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=42] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:50.649841] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=139] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2985, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:50.649940] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=346, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:50.665950] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:50.673924] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A43FF-0-0] [lt=52] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1089) [2025-01-08 00:50:50.674936] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=1509, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:50.674966] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1089) [2025-01-08 00:50:50.675013] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=254, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:50.675247] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=80] [PALF STAT FLUSH CB COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=67, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:50.685789] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:89) [46340][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=19] XA scheduler heartbeat task statistics(avg_time=0) [2025-01-08 00:50:50.696506] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A2-0-0] [lt=14] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250687847, keepalive_ts:-1}) [2025-01-08 00:50:50.696621] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A2-0-0] [lt=112] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250687847, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:50.696766] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A3-0-0] [lt=21] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1001, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250688003, keepalive_ts:-1}) [2025-01-08 00:50:50.696803] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58298][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D1EDBB2A3-0-0] [lt=36] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1001, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250688003, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:50.699082] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.732224] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=16] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:50.734569] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=45] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:721, free_obj_hold_cnt:1, allocator used:2778256, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:1, free_obj_hold_cnt:0, allocator used:65591, allocator total:65591}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():130944, full allocator used:1600, full allocator total:196480}, tablets_mem=3264951, tablets_mem_limit=644245080, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:54, free_obj_hold_cnt:0, allocator used:110592, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=722) [2025-01-08 00:50:50.760768] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [46289][T1_Occam][T1][YB42C0A89F89-00062B2D1B28458E-0-0] [lt=34] ====== tenant freeze timer task ====== [2025-01-08 00:50:50.766376] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=27] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:50.769453] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=155] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2025-01-08 00:50:50.774357] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4401-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=9, avg_log_batch_cnt=0, total_group_log_size=1089, avg_group_log_size=121) [2025-01-08 00:50:50.775584] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=39] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:50.792901] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D002-0-0] [lt=120][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.799227] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.800444] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37EF-0-0] [lt=150] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10347) [2025-01-08 00:50:50.802148] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58342][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0EE-0-0] [lt=34] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250790230, keepalive_ts:-1}) [2025-01-08 00:50:50.802182] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58342][T1_L0_G1][T1][YB42C0A89F8B-00062B2D182BB0EE-0-0] [lt=33] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326250790230, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:50.805830] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=98] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:50.809100] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [50454][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=57] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2ad8f31f42b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-01-08 00:50:50.829587] INFO [LIB] stat (utility.h:1210) [46351][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=99, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:50.835423] INFO [SHARE.LOCATION] vtable_nonblock_renew (ob_vtable_location_service.cpp:446) [46391][BlackListServic][T1][YB42C0A89F89-00062B2D16C84A6E-0-0] [lt=128] add_update_task succeed(ret=0, ret="OB_SUCCESS", task={tenant_id:1, table_id:12239, add_timestamp:1736326250835333}) [2025-01-08 00:50:50.835466] INFO [SHARE.LOCATION] renew_vtable_location_ (ob_vtable_location_service.cpp:207) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594E-0-0] [lt=27] renew vtable location success(ret=0, ret="OB_SUCCESS", tenant_id=1, table_id=12239, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326250835444, sql_renew_time:1736326250835444, is_mark_fail:false}]) [2025-01-08 00:50:50.835506] INFO [SHARE.LOCATION] batch_process_tasks (ob_vtable_location_service.cpp:414) [46192][VTblLocAsyncUp0][T0][YB42C0A89F89-00062B2D1A28594E-0-0] [lt=39] success to process renew task(task={tenant_id:1, table_id:12239, add_timestamp:1736326250835333}, locations=[{table_id:12239, partition_id:12239, partition_cnt:0, replica_locations:[{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, reserved:0, property:{memstore_percent_:100}}], renew_time:1736326250835444, sql_renew_time:1736326250835444, is_mark_fail:false}]) [2025-01-08 00:50:50.839867] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=45] PNIO [ratelimit] time: 1736326250839864, bytes: 866403270, bw: 0.315391 MB/s, add_ts: 1005277, add_bytes: 332457 [2025-01-08 00:50:50.850090] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=49] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:50.852656] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:399) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=30] start to print blacklist info [2025-01-08 00:50:50.853019] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [46391][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=24] ls blacklist refresh finish(cost_time=19770) [2025-01-08 00:50:50.853084] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:504) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5181673537, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:50.853114] WDIAG [LIB] try_wait (thread.cpp:249) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=11, ret=-4023) [2025-01-08 00:50:50.853127] INFO [LIB] do_thread_recycle (threads.cpp:146) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x2ad8885abcf0, thread=0x2ad8d42a0110) [2025-01-08 00:50:50.853136] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:509) [46097][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=8] inc thread count(*this={name:IO_SYNC_CH, this:0x2ad8885abcf0, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:5181673537, tenant_id:500}, cur_thread_count=2, cnt=-1, new_thread_count=1) [2025-01-08 00:50:50.855550] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=108] [PALF STAT WRITE LOG COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=3110, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:50.855601] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=1048, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:50.855689] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT FLUSH CB COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=76, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:50.855927] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=111] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=52, stat_interval=1000000, avg (count/cost)=210, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:50.855964] INFO [LIB] stat (utility.h:1210) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=52, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:50.856618] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}, {id:1003}, {id:1001}, {id:1002}]) [2025-01-08 00:50:50.861209] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:1736326250662489000, v:0}, total_part_count:1, valid_inner_part_count:1, valid_user_part_count:0}, server_version_delta=198638, in_cluster_service=true, cluster_version={val:1736326250456027000, v:0}, min_cluster_version={val:1736326250456027000, v:0}, max_cluster_version={val:1736326250456027000, v:0}, get_cluster_version_err=0, cluster_version_delta=405100, cluster_service_master="192.168.159.137:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=36255, succ_cluster_heartbeat_count=1, cluster_heartbeat_interval=50000, local_cluster_version={val:1736325745503013000, v:0}, local_cluster_delta=505358114, force_self_check=false, weak_read_refresh_interval=100000) [2025-01-08 00:50:50.874107] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C82-0-0] [lt=103][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.879154] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2302) [58298][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4403-0-0] [lt=40][errcode=-4389] log_task life cost too much time(palf_id=1002, self="192.168.159.137:2882", log_id=17870, log_task={header:{begin_lsn:{lsn:2175034}, end_lsn:{lsn:2175156}, log_id:17870, min_scn:{val:1736326250487243002, v:0}, max_scn:{val:1736326250487243002, v:0}, data_len:66, proposal_id:1, prev_lsn:{lsn:2174912}, prev_proposal_id:1, committed_end_lsn:{lsn:2175034}, data_checksum:1715074492, accum_checksum:3397965136, is_padding_log:false, is_raw_write:false}, state_map:{val:58}, ref_cnt:0, gen_ts:1736326250774283, freeze_ts:1736326250774283, submit_ts:1736326250774291, flushed_ts:1736326250777045, gen_to_freeze cost time:0, gen_to_submit cost time:8, submit_to_flush cost time:2754}, fs_cb_begin_ts=1736326250879143, log_life_time=104860) [2025-01-08 00:50:50.881105] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C82-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:50.885034] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:228) [46181][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=93] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-01-08 00:50:50.893450] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=34] ====== check clog disk timer task ====== [2025-01-08 00:50:50.893491] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=34] get_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:50:50.897215] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=36] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.897260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.897596] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.897750] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:50.899414] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=76][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:50.908112] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1736326250908032], gts:1736326250907258000, latest_srr:[mts=1736326250908032]}) [2025-01-08 00:50:50.908159] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=45] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:50.908236] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=57] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1736326250908218], gts:1736326250907258000, latest_srr:[mts=1736326250908218]}) [2025-01-08 00:50:50.908568] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:520) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1736326250908553], gts:1736326250908389000, latest_srr:[mts=1736326250908553]}) [2025-01-08 00:50:50.908991] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=17] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18640215}, readable_log_tail:{lsn:18640215}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48464553, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:50.909097] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=105] get_replay_process success(id={id:1}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:50:50.909112] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1089) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=13] replay status is not follower(min_unreplayed_lsn={lsn:0}, base_lsn={lsn:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324447747312, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436756804000, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad924f71a70, log_storage_:{log_tail:{lsn:2142242}, readable_log_tail:{lsn:2142242}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436756804001, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1001/log", dir_fd:180, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64962526, block_header_serialize_buf_:0x2ad924f72738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:50.909292] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=178] get_replay_process success(id={id:1001}, submitted_log_size=0, unsubmitted_log_size=0, replayed_log_size=0, unreplayed_log_size=0) [2025-01-08 00:50:50.909309] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=12] get_replay_process success(id={id:1002}, submitted_log_size=2175156, unsubmitted_log_size=2175156, replayed_log_size=2175156, unreplayed_log_size=0) [2025-01-08 00:50:50.909321] INFO [CLOG] operator() (ob_log_replay_service.cpp:1679) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=8] get_replay_process success(id={id:1003}, submitted_log_size=2164943, unsubmitted_log_size=2164943, replayed_log_size=2164943, unreplayed_log_size=0) [2025-01-08 00:50:50.909350] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:182) [50415][T1002_TimerWK0][T1002][Y0-0000000000000000-0-0] [lt=7] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=4, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=4, replayed_log_size(MB)=4, last_submitted_log_size(MB)=4, last_replayed_log_size(MB)=4, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-01-08 00:50:50.926282] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=13] ====== check clog disk timer task ====== [2025-01-08 00:50:50.926316] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=28] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:50.932685] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:50.948751] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:50.951527] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=1098) [2025-01-08 00:50:50.952132] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=17] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:50.952259] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=30, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:50.952944] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:50.954140] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1454) [2025-01-08 00:50:50.954535] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18] worker thread began to shrink(tenant_->id()=1002, group_id_=1, token=0) [2025-01-08 00:50:50.955723] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:50.955830] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58298][T1002_L0_G1][T1002][Y0-0000000000000000-0-0] [lt=136] worker thread exit(tenant_->id()=1002, workers_.get_size()=8) [2025-01-08 00:50:50.970664] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:728) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=1069504, replace_num=15728) [2025-01-08 00:50:50.970754] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=88] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2025-01-08 00:50:50.977651] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17784, miss_cnt:0, cache_read_size:2175400, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:50.994161] INFO [SHARE] get_zone_list_ (ob_zone_table_operation.cpp:559) [46301][T1_HBService][T1][YB42C0A89F89-00062B2D18F84D2E-0-0] [lt=18] get inactive zone_list(ret=0, ret="OB_SUCCESS", zone_list=[]) [2025-01-08 00:50:50.996991] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=86, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:50.999548] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.001453] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:51.001520] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44917, [delay/us]:prepare:8, schedule:109, submit:1612, rt:1304, total:3035) [2025-01-08 00:50:51.001537] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:51.001592] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=39] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44923, [delay/us]: prepare:8, schedule:109, submit:1612, device:1304, total:3035) [2025-01-08 00:50:51.001605] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:51.001644] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4864, iops:16, bw:77594, [delay/us]:prepare:26, schedule:135, submit:589, rt:3187, total:3938) [2025-01-08 00:50:51.001654] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=4, iops=4, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:51.001689] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4864, avg_iops:16, avg_bw:77595, [delay/us]: prepare:26, schedule:135, submit:589, device:3187, total:3938) [2025-01-08 00:50:51.001700] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:51.001722] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:2, bw:8167, [delay/us]:prepare:4, schedule:4196, submit:1, rt:1962, total:6164) [2025-01-08 00:50:51.001744] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4300, iops:40, bw:171525, [delay/us]:prepare:7, schedule:149, submit:595, rt:1493, total:2245) [2025-01-08 00:50:51.001753] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:51.001780] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:2, avg_bw:8168, [delay/us]: prepare:4, schedule:4196, submit:1, device:1962, total:6164) [2025-01-08 00:50:51.001796] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4301, avg_iops:40, avg_bw:171534, [delay/us]: prepare:7, schedule:149, submit:595, device:1493, total:2245) [2025-01-08 00:50:51.001806] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:51.001824] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4084, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:1006, total:1017) [2025-01-08 00:50:51.001951] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=105] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:51.001991] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=34] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4083, [delay/us]: prepare:0, schedule:0, submit:600000000, device:1006, total:1017) [2025-01-08 00:50:51.002011] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:51.002029] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:51.002038] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:51.002045] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:51.002053] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SERVER] net_in=316kB/s, net_out=255kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:51.031130] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=14] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=3540) [2025-01-08 00:50:51.048653] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28F-0-0] [lt=117] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:51.048723] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C28F-0-0] [lt=50] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326250487243004, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326250487243004, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:51.048823] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:51.048850] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:51.051123] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=91] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=3897, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:51.051178] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=995, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:51.051341] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=228] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:51.054622] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=43] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.054644] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58333][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36CB-0-0] [lt=51] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1332) [2025-01-08 00:50:51.058311] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=27] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:51.058949] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=28] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8553, offset=2425991}, end_cursor:ObLogCursor{file_id=1, log_id=8554, offset=2426296}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894694050, len:2937, log_data_len:305, seq:8553, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:51.059590] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3C-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:51.059637] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3C-0-0] [lt=28] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326250908389000, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326250908389000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:51.066394] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50624][T1001_L0_G0][T1001][YB42C0A89F8A-00062B2D1A79C28F-0-0] [lt=34] get number(ret=0, service_type_=0, range=1, base_id=1736326251060618000, start_id=1736326251060618001, end_id=1736326251060618002) [2025-01-08 00:50:51.070146] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:51.072241] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=34] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8554, offset=2426296}, end_cursor:ObLogCursor{file_id=1, log_id=8555, offset=2426593}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad894698050, len:2632, log_data_len:297, seq:8554, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:51.072673] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3C-0-0] [lt=23] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:51.072793] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3C-0-0] [lt=104] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326250908389000, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326250908389000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:51.080802] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=41] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:51.086037] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8555, offset=2426593}, end_cursor:ObLogCursor{file_id=1, log_id=8556, offset=2426890}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad89469c050, len:2335, log_data_len:297, seq:8555, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:51.087892] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8556, offset=2426890}, end_cursor:ObLogCursor{file_id=1, log_id=8557, offset=2427187}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946a0050, len:2038, log_data_len:297, seq:8556, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:51.088069] INFO [PALF] next (palf_iterator.h:183) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=123] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17698, miss_cnt:0, cache_read_size:2165065, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:51.099670] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.100551] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50497][T1002_L5_G0][T1002][YB42C0A89F8B-00062B2D1B5BBA47-0-0] [lt=80] get number(ret=0, service_type_=0, range=1, base_id=1736326251099248000, start_id=1736326251099248000, end_id=1736326251099248001) [2025-01-08 00:50:51.133265] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=65] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:51.134874] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:671) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=79] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=377484, clean_num=31457) [2025-01-08 00:50:51.147695] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=27] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:51.154037] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36CD-0-0] [lt=31] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=17, avg_log_submit_to_flush_time=4570, avg_log_submit_to_slide_time=31697) [2025-01-08 00:50:51.154152] INFO [PALF] read (log_cache.cpp:103) [50481][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=21] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1332, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.155082] INFO [PALF] inner_write_once_ (log_block_handler.cpp:410) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=96] [PALF STAT WRITE LOG INFO TO DISK](ret=0, offset=2169283, this={dio_aligned_buf:{buf_write_offset:2621, buf_padding_size:1475, align_size:4096, aligned_buf_size:3682304, aligned_used_ts:23779, truncate_used_ts:5345}, log_block_size:67108864, io_fd:{first_id:-1, second_id:191, third_id:-1, fd_id:-1, slot_version:-1, device_handle:0x2ad886398080}}, aligned_buf_len=4096, aligned_block_offset=2166784, buf_len=122, total_write_size=12172, total_write_size_after_dio=421888, ob_pwrite_used_ts=253471, count=100, each_pwrite_cost=2534) [2025-01-08 00:50:51.155708] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:300) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=76] ====== check clog disk timer task ====== [2025-01-08 00:50:51.155738] INFO [PALF] get_disk_usage (palf_env_impl.cpp:920) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=4608, used(MB):=279) [2025-01-08 00:50:51.169009] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC36B-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.170824] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2025-01-08 00:50:51.174305] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4409-0-0] [lt=9] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=15, this=0x2ad92435f6a0, extra_info="palf_id:1002") [2025-01-08 00:50:51.174396] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4409-0-0] [lt=90] [PALF STAT LOG LIFE TIME](cur_stat_count=9, stat_interval=1000000, avg (count/cost)=34884, this=0x2ad92435f8d0, extra_info="palf_id:1002") [2025-01-08 00:50:51.174384] INFO [PALF] read (log_cache.cpp:103) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT HOT CACHE HIT RATE](palf_id=1002, read_size=1098, hit_cnt=9, read_cnt=9, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.174411] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4409-0-0] [lt=13] [PALF STAT LOG TASK TIME](palf_id=1002, self="192.168.159.137:2882", total_slide_log_cnt=9, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=14, avg_log_submit_to_flush_time=1978, avg_log_submit_to_slide_time=34869) [2025-01-08 00:50:51.174600] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:312K/72:110736 disk:312K/72:110736 t1m2:44K/10:91587 t1m2g0:44K/10:91587 t1002m2:192K/45:124091 t1002m2g0:192K/45:124091 t1001m2:76K/17:86648 t1001m2g0:76K/17:86648 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:51.197270] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AB8F-0-0] [lt=18] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.742941093444824219e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=387, Throttle Trigger(MB)=232, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.809489877078427753e+02) [2025-01-08 00:50:51.197735] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=48] [PALF STAT APPEND COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:51.199724] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.202506] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50917][T1002_L0_G5][T1002][YB42C0A89F8B-00062B2D183BE7CF-0-0] [lt=57] get number(ret=0, service_type_=0, range=1, base_id=1736326251201587000, start_id=1736326251201587000, end_id=1736326251201587001) [2025-01-08 00:50:51.204103] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46476][T1_L0_G5][T1][YB42C0A89F8B-00062B2D183BE7CD-0-0] [lt=24] get number(ret=0, service_type_=0, range=1, base_id=1736326251203917000, start_id=1736326251203917000, end_id=1736326251203917001) [2025-01-08 00:50:51.212746] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46374][T1_L5_G0][T1][YB42C0A89F8B-00062B2D171BBD5D-0-0] [lt=60] get number(ret=0, service_type_=0, range=1, base_id=1736326251211763000, start_id=1736326251211763000, end_id=1736326251211763001) [2025-01-08 00:50:51.214891] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=23] gc stale ls task succ [2025-01-08 00:50:51.222157] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:51.222300] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=140][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:51.227012] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87013-0-0] [lt=7][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.227073] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:51.227085] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:51.246186] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB46-0-0] [lt=26][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193756160017, cache_obj->added_lc()=false, cache_obj->get_object_id()=13552, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:51.247706] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:449) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=27] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:51.247731] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:269) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=24] try reload schema success(schema_version=1736326180221088, refresh_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_ret=0) [2025-01-08 00:50:51.247781] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:263) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E1-0-0] [lt=14] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:51.247800] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:288) [46157][SerScheQueue1][T0][YB42C0A89F89-00062B2D198848E1-0-0] [lt=16] [REFRESH_SCHEMA] local schema info is newer, no need to refresh schema(ret=0, ret="OB_SUCCESS", local_schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}, schema_info={schema_version:1736326241712064, tenant_id:1002, sequence_id:1099511630222}) [2025-01-08 00:50:51.263993] WDIAG read_and_convert_to_tuples_ (ob_table_access_helper.h:451) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4008] iter failed(iter_times=0, ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:50:51.264278] WDIAG read_multi_row (ob_table_access_helper.h:238) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4008] fail to read and convert to tuples(ret=-4008, ret="OB_ITER_END", MTL_ID()=1002, table=oceanbase.__all_detect_lock_info, condition=WHERE detect_func_no = 1 GROUP BY owner_id) [2025-01-08 00:50:51.265452] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=11, avg_log_batch_cnt=1, total_group_log_size=4549, avg_group_log_size=413) [2025-01-08 00:50:51.265485] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=4085) [2025-01-08 00:50:51.265503] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8aec84050, extra_info="id:1") [2025-01-08 00:50:51.266004] INFO delete_row (ob_table_access_helper.h:339) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18] GCTX.sql_proxy_ insert row success(ret=0, ret="OB_SUCCESS", MTL_ID()=1002, table=oceanbase.__all_dbms_lock_allocated, value=expiration <= usec_to_time(1736326251264298) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10, sql=DELETE FROM oceanbase.__all_dbms_lock_allocated WHERE expiration <= usec_to_time(1736326251264298) AND lockid NOT IN (SELECT obj_id FROM oceanbase.__all_detect_lock_info where obj_type = 13 or obj_type = 11)LIMIT 10) [2025-01-08 00:50:51.266079] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-01-08 00:50:51.266135] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] finish check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-01-08 00:50:51.266186] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:349) [50446][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7] finish check and clear obj lock(ls->get_ls_id()={id:1003}) [2025-01-08 00:50:51.266555] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1656) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=17] skip inspect bad block(last_check_time=1736322505277175, last_macro_idx=-1) [2025-01-08 00:50:51.278054] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:50:51.278236] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50566][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%", tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-01-08 00:50:51.279325] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:690) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] no need to update min/max version(new_version={val:1736326250406804004, v:0}, min_version={val:1736326250406804004, v:0}, max_version={val:1736326250406804004, v:0}) [2025-01-08 00:50:51.279959] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:955) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4018] ls not exist in __all_ls_status table(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2025-01-08 00:50:51.280104] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [50423][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4018] fail to inner get ls status info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%", tenant_id=1002, exec_tenant_id=1001, need_member_list=false) [2025-01-08 00:50:51.281317] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50595][T1001_LSArchive][T1001][YB42C0A89F89-00062B2D1CA8458E-0-0] [lt=14] gc stale ls task succ [2025-01-08 00:50:51.281373] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:51.281396] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] temporary file module use no blocks [2025-01-08 00:50:51.281405] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:51.281418] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=4048, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:51.281433] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50443][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193755193600, flush_io_finished_ret_:0, flush_io_finished_round_:1914, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:51.281519] INFO [SHARE] load_can_execute_task (ob_balance_task_table_operator.cpp:842) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C99-0-0] [lt=13] load can-execute balance task(ret=0, ret="OB_SUCCESS", task_array=[], sql=select * from __all_balance_task where parent_list is null or parent_list = '') [2025-01-08 00:50:51.281543] INFO [BALANCE] do_work (ob_balance_task_execute_service.cpp:172) [50390][T1002_BalanceEx][T1002][YB42C0A89F89-00062B2D24984C99-0-0] [lt=22] [BALANCE_EXECUTE] finish one round(ret=0, ret="OB_SUCCESS", task_array_=[], idle_time_us=1000000, task_comment_=) [2025-01-08 00:50:51.293675] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [46278][T1_COMMONLSSe][T1][YB42C0A89F89-00062B2D1918540E-0-0] [lt=17][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.299758] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=93] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:51.299795] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.308830] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58341][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF25-0-0] [lt=42] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326251301281, keepalive_ts:-1}) [2025-01-08 00:50:51.308975] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58341][T1001_L0_G1][T1001][YB42C0A89F8B-00062B2D1CDBAF25-0-0] [lt=142] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.139:2882", region:"sys_region", register_time_us:1736326251301281, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:51.319534] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46212][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.319573] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46211][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.330382] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [46524][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=45] current monitor number(seq_=-1) [2025-01-08 00:50:51.335687] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=94] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:51.350335] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=84, this=0x2ad8fc455470, extra_info="") [2025-01-08 00:50:51.353976] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [46344][T1_LSArchiveMgr][T1][YB42C0A89F89-00062B2D1648458E-0-0] [lt=16] gc stale ls task succ [2025-01-08 00:50:51.370963] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2025-01-08 00:50:51.374353] INFO [CLOG] push_log_replay_task (ob_replay_status.cpp:1158) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=28] submit pre barrier log success(log_type=17, ls_id={id:1002}, lsn={lsn:2175822}, scn={val:1736326250987796003, v:0}, is_pre_barrier=true, is_post_barrier=false, log_size=24) [2025-01-08 00:50:51.374555] INFO [STORAGE] inner_update_reserved_snapshot (ob_ls_reserved_snapshot_mgr.cpp:293) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=46] success to update reserved snapshot(ret=0, ls_id={id:1002}, min_reserved_snapshot_=1736324445564663000) [2025-01-08 00:50:51.374612] INFO [CLOG] do_replay_task_ (ob_log_replay_service.cpp:1002) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=53] pre barrier log replay succ(replay_task={ls_id_:{id:1002}, log_type:17, log_type:"RESERVED_SNAPSHOT", lsn_:{lsn:2175822}, scn_:{val:1736326250987796003, v:0}, is_pre_barrier_:true, is_post_barrier_:false, read_log_size_:24, replay_hint_:1736326251423117, first_handle_ts_:1736326251374538, replay_cost_:72, retry_cost_:-1, read_log_buf_:0x2ad90fe04160, decompression_buf_:null, has_decompressed_:false, decompressed_log_size_:0, base_header_len_:16}, replay_status={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:1, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326251374188, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2175878}, next_to_submit_scn_:{val:1736326250987796004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:112, curr_read_buf_start_pos:0, curr_read_buf_end_pos:112, log_storage_:{start_lsn:{lsn:2175766}, end_lsn:{lsn:2175878}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2175878}, readable_log_tail:{lsn:2175878}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64928890, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326250987796003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:24, scn_:{val:1736326250987796003, v:0}, data_checksum:796858218, flag:63504}}, init_mode_version:0, accumulate_checksum:2096116064, curr_entry_is_padding:0, padding_entry_size:56, padding_entry_scn:{val:1736326250987796003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17788, miss_cnt:0, cache_read_size:2175878, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}, replay_queue_idx=13, ret=0) [2025-01-08 00:50:51.383559] INFO [LIB] stat (utility.h:1210) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=12] [PALF STAT WRITE LOG COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=2168, this=0x2ad9083f9c30, extra_info="palf_id:1") [2025-01-08 00:50:51.383591] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=30] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=10185) [2025-01-08 00:50:51.383624] INFO [LIB] stat (utility.h:1227) [50568][T1001_IOWorker][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=266, this=0x2ad8f97ee530, extra_info="") [2025-01-08 00:50:51.383787] INFO [LIB] stat (utility.h:1210) [50567][T1001_LogIOCb0][T1001][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=31, this=0x2ad9083f9e60, extra_info="palf_id:1") [2025-01-08 00:50:51.385406] INFO [LIB] stat (utility.h:1210) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37F9-0-0] [lt=68] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=22, this=0x2ad9083ef6a0, extra_info="palf_id:1") [2025-01-08 00:50:51.385430] INFO [LIB] stat (utility.h:1210) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37F9-0-0] [lt=23] [PALF STAT LOG LIFE TIME](cur_stat_count=18, stat_interval=1000000, avg (count/cost)=4323, this=0x2ad9083ef8d0, extra_info="palf_id:1") [2025-01-08 00:50:51.385440] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58344][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA37F9-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=18, avg_log_gen_to_freeze_time=130, avg_log_gen_to_submit_time=211, avg_log_submit_to_flush_time=2773, avg_log_submit_to_slide_time=4112) [2025-01-08 00:50:51.386044] INFO [PALF] read (log_cache.cpp:103) [50570][T1001_LogShared][T1001][Y0-0000000000000000-0-0] [lt=47] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=10185, hit_cnt=18, read_cnt=18, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.386171] INFO [STORAGE.TRANS] persist_version_if_need_ (ob_tenant_weak_read_cluster_service.cpp:338) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] persist CLUSTER version range succ(tenant_id=1002, new_min_version={val:1736326250701549003, v:0}, new_max_version={val:1736326250701549003, v:0}, last_min_version={val:1736326250406804004, v:0}, last_max_version={val:1736326250406804004, v:0}, total_time_us=6706, delta_us=684620) [2025-01-08 00:50:51.386403] INFO [STORAGE.TRANS] update_cluster_version (ob_tenant_weak_read_cluster_service.cpp:697) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=227] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] update version(tenant_id=1002, current_version={val:1736326250701549003, v:0}, min_version={val:1736326250701549003, v:0}, max_version={val:1736326250701549003, v:0}, in_service=true, leader_epoch=1) [2025-01-08 00:50:51.390223] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=28] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:51.390275] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=44] temporary file module use no blocks [2025-01-08 00:50:51.390355] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=78] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:51.390428] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=69] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=1518, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:51.390469] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [50582][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=40] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193755303903, flush_io_finished_ret_:0, flush_io_finished_round_:1910, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:51.395170] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:179) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-01-08 00:50:51.395252] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:821) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=78] temporary file module use no blocks [2025-01-08 00:50:51.395261] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:125) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-01-08 00:50:51.395299] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1519) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=36] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=2783, fat_actual_size=253, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-01-08 00:50:51.395313] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:308) [46328][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:193755308868, flush_io_finished_ret_:0, flush_io_finished_round_:3738, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-01-08 00:50:51.399901] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.406217] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E5-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}) [2025-01-08 00:50:51.406272] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8B-00062B2D1A8BB5E5-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326251201587001, v:0}, arg={tenant_id:1002, ls_id:{id:1003}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1003}, cur_readable_scn:{val:1736326251201587001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:51.408926] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=18] get number(ret=0, service_type_=0, range=1, base_id=1736326251408655000, start_id=1736326251408655000, end_id=1736326251408655001) [2025-01-08 00:50:51.418229] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50622][T1001_L0_G0][T1001][YB42C0A89F8B-00062B2D1A8BB5E5-0-0] [lt=25] get number(ret=0, service_type_=0, range=1, base_id=1736326251417174000, start_id=1736326251417174000, end_id=1736326251417174001) [2025-01-08 00:50:51.422006] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [50573][T1001_ApplySrv0][T1001][Y0-0000000000000000-0-0] [lt=13] rpc statistics(total_trans_msg_count=2, total_batch_msg_count=0) [2025-01-08 00:50:51.430229] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=47] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:999, free_obj_hold_cnt:109, allocator used:4263584, allocator total:4569600}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:109, free_obj_hold_cnt:0, allocator used:7149419, allocator total:7149419}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():194432, full allocator used:1600, full allocator total:259968}, tablets_mem=11978987, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:80, free_obj_hold_cnt:40, allocator used:245760, allocator total:261120}, pending_cnt=0, wait_gc_count=0, tablet count=1108) [2025-01-08 00:50:51.434906] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24] worker thread began to shrink(tenant_->id()=1, group_id_=1, token=0) [2025-01-08 00:50:51.435074] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58342][T1_L0_G1][T1][Y0-0000000000000000-0-0] [lt=77] worker thread exit(tenant_->id()=1, workers_.get_size()=2) [2025-01-08 00:50:51.453497] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50559][T1001_TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=47] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=12478464) [2025-01-08 00:50:51.456035] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:51.458637] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2025-01-08 00:50:51.458743] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1685) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=97] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=69206016, ctx.total_memstore_used_=102760448, ctx.total_memstore_hold_=102760448, memstore_freeze_trigger_limit_=70285300, tenant_id=1001) [2025-01-08 00:50:51.458794] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1689) [50545][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=41] [TenantFreezer] oldest frozen memtable(list="") [2025-01-08 00:50:51.459239] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46219][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.459278] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46214][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.459601] INFO [LIB] stat (utility.h:1210) [58306][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36D3-0-0] [lt=96] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=0, this=0x2ad924fd7470, extra_info="") [2025-01-08 00:50:51.471923] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:129) [50400][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=32] ====== tenant freeze timer task ====== [2025-01-08 00:50:51.498928] INFO eloop_run (eloop.c:145) [46145][pnio2][T0][Y0-0000000000000000-0-0] [lt=40] PNIO [ratelimit] time: 1736326251498925, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007748, add_bytes: 0 [2025-01-08 00:50:51.499966] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.520132] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:677) [46217][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.536291] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:51.539821] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=18, avg_log_batch_cnt=1, total_group_log_size=10186, avg_group_log_size=565) [2025-01-08 00:50:51.539917] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50602][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=88] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=10186) [2025-01-08 00:50:51.540528] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:529) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=34] worker thread began to shrink(tenant_->id()=1001, group_id_=1, token=0) [2025-01-08 00:50:51.541494] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:541) [58340][T1001_L0_G1][T1001][Y0-0000000000000000-0-0] [lt=42] worker thread exit(tenant_->id()=1001, workers_.get_size()=8) [2025-01-08 00:50:51.542353] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46213][BatchIO][T0][Y0-0000000000000000-0-0] [lt=26] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.544546] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:642) [46215][BatchIO][T0][Y0-0000000000000000-0-0] [lt=82] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.550782] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=61, this=0x2ad924f6b470, extra_info="") [2025-01-08 00:50:51.550858] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=65] [PALF STAT GROUP LOG INFO](palf_id=1001, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:51.550879] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT APPEND DATA SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, append size=1220) [2025-01-08 00:50:51.550894] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad91ad4a050, extra_info="id:1001") [2025-01-08 00:50:51.553471] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT COMMITTED LOG SIZE](palf_id=1001, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:51.553667] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F6F-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=17, this=0x2ad924f6b6a0, extra_info="palf_id:1001") [2025-01-08 00:50:51.553688] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F6F-0-0] [lt=21] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4184, this=0x2ad924f6b8d0, extra_info="palf_id:1001") [2025-01-08 00:50:51.555078] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F70-0-0] [lt=8] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=16, this=0x2ad8fc4556a0, extra_info="palf_id:1") [2025-01-08 00:50:51.555103] INFO [LIB] stat (utility.h:1210) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F70-0-0] [lt=24] [PALF STAT LOG LIFE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=3582, this=0x2ad8fc4558d0, extra_info="palf_id:1") [2025-01-08 00:50:51.555113] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F70-0-0] [lt=9] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=19, avg_log_gen_to_submit_time=116, avg_log_submit_to_flush_time=2278, avg_log_submit_to_slide_time=3482) [2025-01-08 00:50:51.555616] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=1765, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.555843] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36D5-0-0] [lt=39] [PALF STAT GROUP LOG INFO](palf_id=1003, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=10, avg_log_batch_cnt=0, total_group_log_size=1220, avg_group_log_size=122) [2025-01-08 00:50:51.555905] INFO [LIB] stat (utility.h:1210) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36D5-0-0] [lt=27] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=29, this=0x2ad924fd76a0, extra_info="palf_id:1003") [2025-01-08 00:50:51.555978] INFO [LIB] stat (utility.h:1210) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36D5-0-0] [lt=72] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=31857, this=0x2ad924fd78d0, extra_info="palf_id:1003") [2025-01-08 00:50:51.558750] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=39] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=66551808) [2025-01-08 00:50:51.570328] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:51.571173] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46098][TimerWK0][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2025-01-08 00:50:51.573851] INFO [LIB] stat (utility.h:1210) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4411-0-0] [lt=9] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=12, stat_interval=1000000, avg (count/cost)=0, this=0x2ad92435f470, extra_info="") [2025-01-08 00:50:51.580994] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=122] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:51.584647] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:2176244}, scn={val:1736326251487377001, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326251576313, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:14856, ret_code_:0}}, next_to_submit_lsn_:{lsn:2176244}, next_to_submit_scn_:{val:1736326251487377001, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436803442000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad91ad99000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2176122}, end_lsn:{lsn:2176244}, read_buf:{buf_len_:73728, buf_:0x2ad91ad99000}, block_size:67104768, log_storage_:0x2ad924365a70, log_storage_:{log_tail:{lsn:2176244}, readable_log_tail:{lsn:2176244}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436803442001, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1002/log", dir_fd:185, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64928524, block_header_serialize_buf_:0x2ad924366738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326251487377000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326251487377000, v:0}, data_checksum:3348837161, flag:7917}}, init_mode_version:0, accumulate_checksum:2050177141, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326251487377000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17791, miss_cnt:0, cache_read_size:2176244, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:51.600253] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.632374] INFO [LIB] stat (utility.h:1210) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=9] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2124, this=0x2ad8aecdfc30, extra_info="palf_id:1") [2025-01-08 00:50:51.632433] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=57] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=3962) [2025-01-08 00:50:51.632457] INFO [LIB] stat (utility.h:1227) [46314][T1_IOWorker][T1][Y0-0000000000000000-0-0] [lt=17] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=354, this=0x2ad89bfed7d0, extra_info="") [2025-01-08 00:50:51.632544] INFO [LIB] stat (utility.h:1210) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=39] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=35, this=0x2ad8aecdfe60, extra_info="palf_id:1") [2025-01-08 00:50:51.633715] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA672C-0-0] [lt=47] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=21, this=0x2ad8aecd56a0, extra_info="palf_id:1") [2025-01-08 00:50:51.633736] INFO [LIB] stat (utility.h:1210) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA672C-0-0] [lt=20] [PALF STAT LOG LIFE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=4963, this=0x2ad8aecd58d0, extra_info="palf_id:1") [2025-01-08 00:50:51.633746] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58332][T1_L0_G1][T1][YB42C0A89F8A-00062B2D17FA672C-0-0] [lt=8] [PALF STAT LOG TASK TIME](palf_id=1, self="192.168.159.137:2882", total_slide_log_cnt=10, avg_log_gen_to_freeze_time=8, avg_log_gen_to_submit_time=111, avg_log_submit_to_flush_time=3554, avg_log_submit_to_slide_time=4852) [2025-01-08 00:50:51.633826] INFO [PALF] read (log_cache.cpp:103) [46316][T1_LogSharedQue][T1][Y0-0000000000000000-0-0] [lt=89] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=3962, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:51.647927] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46306][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:51.653826] INFO [LIB] stat (utility.h:1210) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=82] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2329, this=0x2ad924f75c30, extra_info="palf_id:1001") [2025-01-08 00:50:51.653861] INFO [LIB] stat (utility.h:1227) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=246, this=0x2ad8f0ffec70, extra_info="") [2025-01-08 00:50:51.654852] INFO [RPC.OBRPC] th_init (ob_rpc_translator.cpp:33) [58346][][T1002][Y0-0000000000000000-0-0] [lt=0] Init thread local success [2025-01-08 00:50:51.655366] INFO [SERVER.OMT] check_worker_count (ob_tenant.cpp:516) [46262][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=88] worker thread created(tenant_->id()=1002, group_id_=1, token=8) [2025-01-08 00:50:51.675143] INFO [LIB] stat (utility.h:1210) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT WRITE LOG COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=1729, this=0x2ad924369c30, extra_info="palf_id:1002") [2025-01-08 00:50:51.675234] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=87] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1332) [2025-01-08 00:50:51.675258] INFO [LIB] stat (utility.h:1227) [50427][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=19] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=134, this=0x2ad8f0fff3b0, extra_info="") [2025-01-08 00:50:51.675372] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=42] [PALF STAT FLUSH CB COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=76, this=0x2ad924369e60, extra_info="palf_id:1002") [2025-01-08 00:50:51.677234] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58329][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D623-0-0] [lt=12] [PALF STAT COMMITTED LOG SIZE](palf_id=1002, self="192.168.159.137:2882", committed size=1454) [2025-01-08 00:50:51.679944] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=175] get_max_applied_scn(scn={val:1736326251408655001, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:1, palf_committed_end_lsn_:{lsn:18641614}, palf_committed_end_scn_:{val:1736326251408655002, v:0}, last_check_scn_:{val:1736326251408655002, v:0}, max_applied_cb_scn_:{val:1736326251408655001, v:0}}) [2025-01-08 00:50:51.680003] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1024) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=55] get_min_unreplayed_log_info in skip state(lsn={lsn:0}, scn={val:1, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736324399616546, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:0}, next_to_submit_scn_:{val:0, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1, v:0}, iterator_:{iterator_impl:{buf_:null, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{start_lsn:{lsn:0}, end_lsn:{lsn:0}, read_buf:{buf_len_:0, buf_:null}, block_size:67104768, log_storage_:0x2ad8fc45ba70, log_storage_:{log_tail:{lsn:18641614}, readable_log_tail:{lsn:18641614}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:2, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1/log", dir_fd:178, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:48463154, block_header_serialize_buf_:0x2ad8fc45c738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:51.680112] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50460][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=87] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:50:51.695457] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:817) [50560][T1001_TimerWK2][T1001][Y0-0000000000000000-0-0] [lt=26] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):105, sequence:1}) [2025-01-08 00:50:51.700376] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.705832] INFO [OCCAM] get_idx (ob_occam_time_guard.h:224) [58346][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E9A8F75-0-0] [lt=39] init point thread id with(&point=0x5572d50e23c0, idx_=1002, point=[thread id=58346, timeout ts=16:00:00.0, last click point="(null):(null):0", last click ts=16:00:00.0], thread_id=58346) [2025-01-08 00:50:51.736759] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=61] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6750040064, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:51.766470] INFO do_work (ob_rl_mgr.cpp:710) [46155][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=45] swc wakeup.(stat_period_=1000000, ready=false) [2025-01-08 00:50:51.767294] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:801) [46098][TimerWK0][T0][YB42C0A89F89-00062B2D1718510F-0-0] [lt=86] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.159.137:2882", "192.168.159.138:2882", "192.168.159.139:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1736322126}, location:{server:"192.168.159.137:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1736322126}, location:{server:"192.168.159.138:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}, {key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1736322126}, location:{server:"192.168.159.139:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}}]) [2025-01-08 00:50:51.771444] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=52] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2025-01-08 00:50:51.772769] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:243) [46265][T1_Occam][T1][YB42C0A89F89-00062B2D17085141-0-0] [lt=26] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:51.774267] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [58337][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D208A4415-0-0] [lt=27] [PALF STAT GROUP LOG INFO](palf_id=1002, self="192.168.159.137:2882", role="FOLLOWER", total_group_log_cnt=11, avg_log_batch_cnt=0, total_group_log_size=1332, avg_group_log_size=121) [2025-01-08 00:50:51.775958] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [46350][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=30] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:51.789612] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C49D005-0-0] [lt=50][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.800476] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.803139] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58341][T1001_L0_G1][T1001][YB42C0A89F8A-00062B2D1CBA3801-0-0] [lt=23] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=10186) [2025-01-08 00:50:51.806119] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50601][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=26] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-01-08 00:50:51.827892] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:167) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=100] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-01-08 00:50:51.827940] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:201) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=38] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-01-08 00:50:51.827916] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=46] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:51.827958] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:295) [46305][T1_TimerWK1][T1][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=190) [2025-01-08 00:50:51.827969] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [46307][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=46] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=175) [2025-01-08 00:50:51.830342] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [50572][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=79] LogLoopThread round_cost_time(us)(round_cost_time=14) [2025-01-08 00:50:51.841423] INFO eloop_run (eloop.c:145) [46141][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1736326251841420, bytes: 866618306, bw: 0.204756 MB/s, add_ts: 1001556, add_bytes: 215036 [2025-01-08 00:50:51.853289] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=54] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=2066) [2025-01-08 00:50:51.856852] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:196) [50467][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1}, {id:1003}]) [2025-01-08 00:50:51.859409] INFO [LIB] stat (utility.h:1210) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=59] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=1536, this=0x2ad924fe1c30, extra_info="palf_id:1003") [2025-01-08 00:50:51.859441] INFO [LIB] stat (utility.h:1227) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=27] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=118, this=0x2ad8f0fffaf0, extra_info="") [2025-01-08 00:50:51.859516] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=67, this=0x2ad924fe1e60, extra_info="palf_id:1003") [2025-01-08 00:50:51.859907] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:458) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] [COMPACTION DAG STATUS] (start_time=1736326241846965, end_time=1736326251859905, dag_status={}) [2025-01-08 00:50:51.859958] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=46] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.859975] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=17] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.859983] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.859991] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.859998] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.860005] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.860011] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.860017] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.860121] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2861) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=102] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-01-08 00:50:51.860135] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860148] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860156] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE/MEDIUM_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860171] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_BATCH_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860180] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_PREPARE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860188] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_SCHEDULE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860197] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"CO_MERGE_FINISH", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860446] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=248] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860462] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860471] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"VERIFY_CKM", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860479] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:23, dag_type_str:"BATCH_FREEZE", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860488] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"UPDATE_SKIP_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860496] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"REFRESH_MAJOR", dag_module_str:"COMPACTION"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860590] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=93] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_COMPLEMENT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860603] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860611] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860619] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860627] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL_DEL_LOB_META", dag_module_str:"DROP_VEC_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860635] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860642] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:6, sys_task_type:25, dag_type_str:"DDL_TABLET_SPLIT", dag_module_str:"DDL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860649] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860755] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=104] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATE_WARMUP", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860770] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=15] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"WAIT_DATA_READY", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860778] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860785] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860793] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860800] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860829] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860841] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860849] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860857] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860864] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860872] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860879] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860886] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_CHECKE_CONVERT", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860893] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860906] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860915] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860923] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860931] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860938] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860946] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860958] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860967] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860975] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860983] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860991] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS_GROUP", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.860998] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG_LS", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861005] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"INITIAL_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861013] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"START_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861025] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_TABLET_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861034] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"FINISH_BACKUP_FUSE", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861042] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861049] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861057] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861064] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861076] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861084] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861092] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861100] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861107] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861114] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861124] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861135] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861143] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861151] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861158] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861167] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861174] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BUILD_TABLET_INFO", dag_module_str:"TRANSFER"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861181] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861240] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=57] dump_dag_status(type={init_dag_prio:3, sys_task_type:21, dag_type_str:"TENANT_SNAPSHOT_CREATE", dag_module_str:"TSNAP_CR8"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861253] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:5, sys_task_type:22, dag_type_str:"TENANT_SNAPSHOT_GC", dag_module_str:"TSNAP_GC"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861282] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4328) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=28] dump_dag_status(type={init_dag_prio:8, sys_task_type:24, dag_type_str:"VECTOR_INDEX_DAG", dag_module_str:"VECTOR_INDEX"}, dag_count=0, running_dag_count=0, added_dag_count=0, scheduled_dag_count=0, scheduled_task_count=0, scheduled_data_size=0) [2025-01-08 00:50:51.861295] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3635) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-01-08 00:50:51.861303] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_count=0) [2025-01-08 00:50:51.861310] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=7] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_count=0) [2025-01-08 00:50:51.861315] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_count=0) [2025-01-08 00:50:51.861321] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_count=0) [2025-01-08 00:50:51.861326] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_count=0) [2025-01-08 00:50:51.861330] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_count=0) [2025-01-08 00:50:51.861334] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_count=0) [2025-01-08 00:50:51.861339] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=5] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_CO_MAJOR", dag_count=0) [2025-01-08 00:50:51.861343] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3638) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_count=0) [2025-01-08 00:50:51.861348] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4342) [50554][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-01-08 00:50:51.868423] INFO [LIB] stat (utility.h:1210) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=11, stat_interval=1000000, avg (count/cost)=79, this=0x2ad8aecd5470, extra_info="") [2025-01-08 00:50:51.871328] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [46343][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=true, can_update_version=true, cur_leader_epoch=1, start_service_tstamp_=1736322551378930, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-01-08 00:50:51.876667] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=11] [REPLAY STAT REPLAY TASK IN QUEUE TIME](cur_stat_count=53, stat_interval=1000000, avg (count/cost)=128, this=0x2ad8f30b28f8, extra_info="") [2025-01-08 00:50:51.877265] INFO [LIB] stat (utility.h:1210) [50482][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=596] [REPLAY STAT REPLAY TASK EXECUTE COST TIME](cur_stat_count=53, stat_interval=1000000, avg (count/cost)=6, this=0x2ad8f30b2b28, extra_info="") [2025-01-08 00:50:51.890990] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1][YB42C0A89F89-00062B2D24784C83-0-0] [lt=47][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.892957] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50534][T1001_COMMONLSS][T1001][YB42C0A89F89-00062B2D24784C83-0-0] [lt=46][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:51.896538] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46206][RpcIO][T0][Y0-0000000000000000-0-0] [lt=84] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.896576] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46208][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.897368] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46209][RpcIO][T0][Y0-0000000000000000-0-0] [lt=26] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.897408] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:604) [46207][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-01-08 00:50:51.900601] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:51.909678] INFO [STORAGE.TRANS] handle_request (ob_timestamp_service.cpp:175) [46205][TsMgr][T1][Y0-0000000000000000-0-0] [lt=47] handle gts request statistics(total_rt=2941, total_cnt=284, avg_rt=1.031929824561403564e+01, avg_cnt=2.839999999999999858e+01) [2025-01-08 00:50:51.909744] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [46205][TsMgr][T0][Y0-0000000000000000-0-0] [lt=51] refresh gts functor(ret=0, ret="OB_SUCCESS", gts_tenant_info={v:1}) [2025-01-08 00:50:51.913797] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:143) [50418][T1002_TimerWK3][T1002][YB42C0A89F89-00062B2D1C58541C-0-0] [lt=50] InfoPoolResizeTask(cost_ts=10) [2025-01-08 00:50:51.924296] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [50617][T1001_L5_G0][T1001][YB42C0A89F8A-00062B2D1C79C275-0-0] [lt=42] get number(ret=0, service_type_=0, range=1, base_id=1736326251923892000, start_id=1736326251923892000, end_id=1736326251923892001) [2025-01-08 00:50:51.937218] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6747942912, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:51.951473] INFO [LIB] stat (utility.h:1210) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] [PALF STAT APPEND COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=0, this=0x2ad8fc4b0050, extra_info="id:1") [2025-01-08 00:50:51.953935] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=2066) [2025-01-08 00:50:51.954061] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=16106127, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=14558208) [2025-01-08 00:50:51.954566] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50426][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:51.954743] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=50] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=20, this=0x2ad924f75e60, extra_info="palf_id:1001") [2025-01-08 00:50:51.956450] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-01-08 00:50:51.957110] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2231) [50428][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, accum_size=1220) [2025-01-08 00:50:51.971609] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2025-01-08 00:50:51.977735] INFO [PALF] next (palf_iterator.h:183) [50434][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1002, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17795, miss_cnt:0, cache_read_size:2176732, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:51.999118] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=60, this=0x2ad9083ef470, extra_info="") [2025-01-08 00:50:52.000855] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=38] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"192.168.159.137:2882", switch_to_leader_ts:-1}) [2025-01-08 00:50:52.000890] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.003994] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1371) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 , local_read=0 0 , local_write=0 0 , remote_read=0 0 , remote_write=0 0 ) [2025-01-08 00:50:52.004046] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS GROUP SYS](tenant_id=1, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4505, iops:10, bw:44942, [delay/us]:prepare:7, schedule:61, submit:53, rt:1217, total:1340) [2025-01-08 00:50:52.004062] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=1, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:3, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:3}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=2, iops=2, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:52.004118] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS FUNCTION](tenant_id=1, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4506, avg_iops:10, avg_bw:44942, [delay/us]: prepare:7, schedule:61, submit:53, device:1217, total:1340) [2025-01-08 00:50:52.004132] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:334, idx_:0, tid_:46309}], io_callback_status=runner 0: thread_id=46309, queue_count=0, ) [2025-01-08 00:50:52.004235] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP SYS](tenant_id=1001, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4527, iops:19, bw:85799, [delay/us]:prepare:10, schedule:67, submit:124, rt:1058, total:1261) [2025-01-08 00:50:52.004252] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=1001, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:1073741824, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=5, iops=5, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:52.004292] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS FUNCTION](tenant_id=1001, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4527, avg_iops:19, avg_bw:85792, [delay/us]: prepare:10, schedule:67, submit:124, device:1058, total:1261) [2025-01-08 00:50:52.004303] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:452, idx_:0, tid_:50563}], io_callback_status=runner 0: thread_id=50563, queue_count=0, ) [2025-01-08 00:50:52.004328] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:SLOG_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4096, iops:4, bw:16341, [delay/us]:prepare:4, schedule:17, submit:0, rt:1471, total:1494) [2025-01-08 00:50:52.004351] INFO [COMMON] print_io_status (ob_io_manager.cpp:2700) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS GROUP SYS](tenant_id=1002, io_status=sys_group_name:CLOG_WRITE_IO, mode:LOCAL WRITE, cur_req:0, hold_mem:0 [FAILED]: fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]: size:4195, iops:41, bw:171584, [delay/us]:prepare:7, schedule:85, submit:346, rt:1136, total:1575) [2025-01-08 00:50:52.004359] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=1002, ref_cnt=2, io_config={group_configs_cnt:3, memory_limit_:3221225472, callback_thread_count_:0, unit_config_:{min_iops:9223372036854775807, max_iops:9223372036854775807, weight:4, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:4}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:0, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=30968256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=10, iops=10, ibw=0, obw=0, iops_limit=9223372036854775807, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:52.004387] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:OTHER_GROUPS, mode:LOCAL WRITE, avg_size:4096, avg_iops:4, avg_bw:16341, [delay/us]: prepare:4, schedule:17, submit:0, device:1471, total:1494) [2025-01-08 00:50:52.004402] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS FUNCTION](tenant_id=1002, io_status=function_name:CLOG_HIGH, mode:LOCAL WRITE, avg_size:4196, avg_iops:41, avg_bw:171584, [delay/us]: prepare:7, schedule:85, submit:346, device:1136, total:1575) [2025-01-08 00:50:52.004412] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:393, idx_:0, tid_:50420}], io_callback_status=runner 0: thread_id=50420, queue_count=0, ) [2025-01-08 00:50:52.004432] INFO [COMMON] print_io_status (ob_io_manager.cpp:2603) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, bw:4085, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:217, total:226) [2025-01-08 00:50:52.004455] INFO [COMMON] print_io_status (ob_io_manager.cpp:2730) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=5752960, free_req_cnt=5000, free_result_cnt=5000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-01-08 00:50:52.004481] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2810) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4085, [delay/us]: prepare:0, schedule:0, submit:600000000, device:217, total:226) [2025-01-08 00:50:52.004499] INFO [COMMON] print_io_status (ob_io_manager.cpp:2752) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:279, idx_:0, tid_:46227}], io_callback_status=runner 0: thread_id=46227, queue_count=0, ) [2025-01-08 00:50:52.004514] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CHANNEL] device_id=0x2ad8885ab2d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:52.004522] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CHANNEL] device_id=0x2ad8885aced0, req_in_channel, async_total=0, sync_total=0, async_cnt=, sync_cnt= 0 [2025-01-08 00:50:52.004530] INFO [COMMON] print_status (ob_io_struct.cpp:2853) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CHANNEL] device_id=0x2ad8885ac0d0, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-01-08 00:50:52.004538] INFO [COMMON] print_server_status (ob_io_manager.cpp:387) [46228][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SERVER] net_in=206kB/s, net_out=219kB/s, bucket_in=0kB/s, bucket_out=0kB/s, failed_bucket_in=0kB/s, failed_bucket_out=0kB/s, limit=128000kB/s [2025-01-08 00:50:52.009906] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46506][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB48-0-0] [lt=81][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193756923739, cache_obj->added_lc()=false, cache_obj->get_object_id()=13553, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78e2f37 0x7ac5a44 0x1040107f 0x10319147 0x78afa91 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:52.017600] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46491][T1_L0_G0][T1][YB42C0A89F8A-00062B2D1979DB4D-0-0] [lt=193] get number(ret=0, service_type_=0, range=1, base_id=1736326252017250000, start_id=1736326252017250000, end_id=1736326252017250001) [2025-01-08 00:50:52.022009] INFO [STORAGE.TRANS] statistics_ (ob_trans_rpc.cpp:678) [46319][T1_ApplySrv0][T1][YB42C0A89F89-00062B2D1B184608-0-0] [lt=7] rpc statistics(total_trans_msg_count=1, total_batch_msg_count=0) [2025-01-08 00:50:52.036969] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1567) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3E-0-0] [lt=44] receive register_req from acceptor, ignore(self="192.168.159.137:2882", palf_id=1, child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326252138826, keepalive_ts:-1}) [2025-01-08 00:50:52.037070] INFO [PALF] handle_register_parent_req (palf_handle_impl.cpp:1586) [58329][T1002_L0_G1][T1002][YB42C0A89F8A-00062B2D1E89BF3E-0-0] [lt=99] handle_register_parent_req finished(ret=0, ret="OB_SUCCESS", palf_id=1, self="192.168.159.137:2882", child={server:"192.168.159.138:2882", region:"sys_region", register_time_us:1736326252138826, keepalive_ts:-1}, is_to_leader=true) [2025-01-08 00:50:52.051541] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1093) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=28] [PALF STAT GROUP LOG INFO](palf_id=1, self="192.168.159.137:2882", role="LEADER", total_group_log_cnt=10, avg_log_batch_cnt=1, total_group_log_size=2066, avg_group_log_size=206) [2025-01-08 00:50:52.051587] INFO [PALF] submit_log (palf_handle_impl.cpp:458) [50468][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, append size=2066) [2025-01-08 00:50:52.053837] INFO [LIB] stat (utility.h:1210) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT WRITE LOG COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=2073, this=0x2ad8fc45fc30, extra_info="palf_id:1") [2025-01-08 00:50:52.053874] INFO [LIB] stat (utility.h:1227) [50425][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=118, this=0x2ad8f0ffe530, extra_info="") [2025-01-08 00:50:52.054661] INFO [LIB] stat (utility.h:1210) [50424][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT FLUSH CB COST TIME](cur_stat_count=10, stat_interval=1000000, avg (count/cost)=54, this=0x2ad8fc45fe60, extra_info="palf_id:1") [2025-01-08 00:50:52.055880] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [58337][T1002_L0_G1][T1002][YB42C0A89F8B-00062B2D21BC36DF-0-0] [lt=121] [PALF STAT COMMITTED LOG SIZE](palf_id=1003, self="192.168.159.137:2882", committed size=1220) [2025-01-08 00:50:52.056359] INFO [PALF] read (log_cache.cpp:103) [50430][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=45] [PALF STAT HOT CACHE HIT RATE](palf_id=1001, read_size=1220, hit_cnt=10, read_cnt=10, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:52.059001] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [50417][T1002_TimerWK2][T1002][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=75665245, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=68631552) [2025-01-08 00:50:52.067431] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:41) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=36] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-01-08 00:50:52.067468] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:107) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=103) [2025-01-08 00:50:52.070638] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1771) [46313][T1_LogIOCb0][T1][Y0-0000000000000000-0-0] [lt=26] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self="192.168.159.137:2882", committed size=4949) [2025-01-08 00:50:52.070719] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=110] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-01-08 00:50:52.076615] INFO [STORAGE] refresh_queuing_mode (ob_tenant_tablet_stat_mgr.cpp:1165) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=12] refresh queuing mode(ret=0, tenant_id=1002, stream_cnt=0, update_schema_cnt=0, cost_time=11) [2025-01-08 00:50:52.076642] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1215) [50418][T1002_TimerWK3][T1002][Y0-0000000000000000-0-0] [lt=25] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1002, interval_step=1, global_iter_pool={is_inited:false, is_washing:false, is_disabled:false, get_cnt:0, tenant_id:1002, bucket_cnt:0, tenant_mem_user_limit:3221225472, tenant_mem_user_hold:243437568, cached_node_array:0x2ad8ba632ee0}) [2025-01-08 00:50:52.079679] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C290-0-0] [lt=26] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}) [2025-01-08 00:50:52.079781] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F8A-00062B2D1A79C290-0-0] [lt=45] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326251986852000, v:0}, arg={tenant_id:1002, ls_id:{id:1002}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1002}, cur_readable_scn:{val:1736326251986852000, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:52.081232] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [50558][T1001_TimerWK0][T1001][Y0-0000000000000000-0-0] [lt=40] Construct Queue Num(construct_num=0, get_push_idx()=10, get_cur_idx()=2, get_pop_idx()=2) [2025-01-08 00:50:52.088012] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:141) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185781-0-0] [lt=49][errcode=0] set logical del time(cache_obj->get_logical_del_time()=193757001844, cache_obj->added_lc()=false, cache_obj->get_object_id()=13554, cache_obj->get_tenant_id()=1, lbt()="0x810107f 0x10bac60e 0x78d12ff 0x78c2d42 0x78b88e4 0x78b1ae0 0x78aefed 0x789e119 0xfc77119 0x1f95655e 0x2ad878b3fea5 0x2ad878e5296d") [2025-01-08 00:50:52.088518] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185782-0-0] [lt=27][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-01-08 00:50:52.089924] WDIAG get_refactored (ob_hashtable.h:1120) [46491][T1_L0_G0][T1][YB42C0A89F89-00062B2D1B185784-0-0] [lt=23][errcode=-4201] hashtable not init [2025-01-08 00:50:52.094565] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=29] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8557, offset=2427187}, end_cursor:ObLogCursor{file_id=1, log_id=8558, offset=2427492}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946a4050, len:1741, log_data_len:305, seq:8557, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.095374] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3D-0-0] [lt=55] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}) [2025-01-08 00:50:52.095427] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3D-0-0] [lt=29] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326251909938001, v:0}, arg={tenant_id:1002, ls_id:{id:1}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1}, cur_readable_scn:{val:1736326251909938001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:52.098601] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:612) [50561][T1001_TimerWK3][T1001][Y0-0000000000000000-0-0] [lt=23] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3788EEE", sizeof(T):3824, used_obj_cnt:992, free_obj_hold_cnt:0, allocator used:3817216, allocator total:4047360}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():178560, full allocator used:1600, full allocator total:244096}, tablets_mem=4291456, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:75, free_obj_hold_cnt:0, allocator used:153600, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=992) [2025-01-08 00:50:52.104051] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.110954] INFO [PALF] check_parent_health (log_config_mgr.cpp:2835) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] re_register_parent reason(palf_id=1002, self="192.168.159.137:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1736326248101805, last_first_register_time_us=1736324449183670, register_time_us=1736326248101805, curr_time_us=1736326252110949) [2025-01-08 00:50:52.111059] INFO [PALF] register_parent_ (log_config_mgr.cpp:2667) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=44] register_parent_(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882", reason=1, leader="192.168.159.138:2882", register_time_us=1736326252110999, last_submit_register_req_time_us=1736326252110999, register_parent_reason=1) [2025-01-08 00:50:52.111080] INFO [PALF] check_parent_health (log_config_mgr.cpp:2844) [50432][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="192.168.159.137:2882") [2025-01-08 00:50:52.111172] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=55] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8558, offset=2427492}, end_cursor:ObLogCursor{file_id=1, log_id=8559, offset=2427789}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946a8050, len:1436, log_data_len:297, seq:8558, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.111635] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3650) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3D-0-0] [lt=24] start get_ls_replayed_scn(arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}) [2025-01-08 00:50:52.111835] INFO [SERVER] get_ls_replayed_scn (ob_service.cpp:3733) [58147][T1002_L0_G18][T1002][YB42C0A89F89-00062B2D1BC84E3D-0-0] [lt=177] finish get_ls_replayed_scn(ret=0, ret="OB_SUCCESS", cur_readable_scn={val:1736326251909938001, v:0}, arg={tenant_id:1002, ls_id:{id:1001}, all_replica:false}, result={tenant_id:1002, ls_id:{id:1001}, cur_readable_scn:{val:1736326251909938001, v:0}, offline_scn:{val:18446744073709551615, v:3}, self_addr_:"192.168.159.137:2882"}, offline_scn={val:18446744073709551615, v:3}) [2025-01-08 00:50:52.120592] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=101] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8559, offset=2427789}, end_cursor:ObLogCursor{file_id=1, log_id=8560, offset=2428086}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946ac050, len:1139, log_data_len:297, seq:8559, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.121556] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=32] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8560, offset=2428086}, end_cursor:ObLogCursor{file_id=1, log_id=8561, offset=2428383}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946b0050, len:842, log_data_len:297, seq:8560, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.121728] INFO [PALF] next (palf_iterator.h:183) [50479][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=111] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17709, miss_cnt:0, cache_read_size:2166407, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}) [2025-01-08 00:50:52.137767] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1136) [46100][TimerWK2][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-6745845760, global_cache_size=343326720, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-01-08 00:50:52.148255] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:52) [46304][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=12, get_cur_idx()=4, get_pop_idx()=4) [2025-01-08 00:50:52.155472] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1042) [50896][T1002_L0_G2][T1002][YB42C0A89F8B-00062B2D1EABD780-0-0] [lt=94] get_min_unreplayed_log_info(lsn={lsn:2166407}, scn={val:1736326251703671004, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1736326252121666, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:2166407}, next_to_submit_scn_:{val:1736326251703671004, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1736324436825152000, v:0}, iterator_:{iterator_impl:{buf_:0x2ad938903000, next_round_pread_size:3678208, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{start_lsn:{lsn:2166285}, end_lsn:{lsn:2166407}, read_buf:{buf_len_:73728, buf_:0x2ad938903000}, block_size:67104768, log_storage_:0x2ad924fdda70, log_storage_:{log_tail:{lsn:2166407}, readable_log_tail:{lsn:2166407}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:0}, min_scn:{val:1736324436825152001, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/ob/data/clog/tenant_1002/1003/log", dir_fd:188, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:64938361, block_header_serialize_buf_:0x2ad924fde738, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1736326251703671003, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1736326251703671003, v:0}, data_checksum:2124447694, flag:40711}}, init_mode_version:0, accumulate_checksum:2484747650, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1736326251703671003, v:0}}, io_ctx:{user:"REPLAY", palf_id:1003, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:17709, miss_cnt:0, cache_read_size:2166407, hit_ratio:"1.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:0, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:0, read_io_size:0, read_disk_cost_ts:0, start_lsn:{lsn:0}}}}}}) [2025-01-08 00:50:52.161625] INFO [PALF] sliding_cb (log_sliding_window.cpp:2291) [58324][T1002_L0_G1][T1002][YB42C0A89F89-00062B2D2228D62C-0-0] [lt=203] [PALF STAT LOG TASK TIME](palf_id=1003, self="192.168.159.137:2882", total_slide_log_cnt=11, avg_log_gen_to_freeze_time=0, avg_log_gen_to_submit_time=19, avg_log_submit_to_flush_time=1647, avg_log_submit_to_slide_time=20387) [2025-01-08 00:50:52.161772] INFO [PALF] read (log_cache.cpp:103) [50477][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=90] [PALF STAT HOT CACHE HIT RATE](palf_id=1003, read_size=1342, hit_cnt=11, read_cnt=11, hit rate=1.000000000000000000e+00) [2025-01-08 00:50:52.164110] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:133) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] ====== checkpoint timer task ====== [2025-01-08 00:50:52.165156] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=101] succ to get rec scn(*this={this:0x2ad90ec04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:2, v:0}, max_applied_start_working_ts_:{val:2, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324393986089, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:713}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.165277] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=119] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc596080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc596080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.165462] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:265) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=54] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1736324427835279009, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, }, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.165629] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=163] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2ad8fc4b4ad0}, min_rec_scn={val:1736324405796530000, v:0}, ls_id_={id:1}) [2025-01-08 00:50:52.165652] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-01-08 00:50:52.165666] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:1736326243478874001, v:0}) [2025-01-08 00:50:52.165673] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:308) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=1, rec_log_ts={val:1736324403874012002, v:0}) [2025-01-08 00:50:52.165686] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=4] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:52.165726] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=34] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-01-08 00:50:52.165826] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:189) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1736324403874012002, v:0}, checkpoint_scn_in_ls_meta={val:1736324403874012002, v:0}, ls_id={id:1}, service_type="TRANS_ID") [2025-01-08 00:50:52.165872] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:52.165895] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1}) [2025-01-08 00:50:52.166795] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad90ecb6030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"L_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"T_APPLY_PENDING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436756804001, v:0}, max_applied_start_working_ts_:{val:1736324436756804001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324441053872, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.166853] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=58] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc597350, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc597350, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.166900] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=23] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246900070001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.166963] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=61] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-01-08 00:50:52.166977] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436756804000, v:0}) [2025-01-08 00:50:52.166991] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:50:52.167005] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-01-08 00:50:52.167021] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2143828}, scn={val:1736326251909938001, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:52.169695] INFO [PALF] pread (log_reader.cpp:130) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF STAT READ LOG INFO FROM DISK](accum_read_io_count=2, accum_read_log_size=8192, avg_pread_cost=1682) [2025-01-08 00:50:52.170610] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=27] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8561, offset=2428383}, end_cursor:ObLogCursor{file_id=1, log_id=8562, offset=2428680}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946b4050, len:545, log_data_len:297, seq:8561, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.170735] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326251909938001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:52.170766] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436756804000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:52.170796] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=30] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1001}) [2025-01-08 00:50:52.171641] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x2ad8dc204030, ls_id:{id:1002}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1002}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436803442001, v:0}, max_applied_start_working_ts_:{val:1736324436803442001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324443551637, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.171728] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=86] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc598620, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc598620, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.171772] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246984891000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.171886] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=111] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1002}) [2025-01-08 00:50:52.171899] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1002}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436803442000, v:0}) [2025-01-08 00:50:52.171913] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:50:52.171969] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=53] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-01-08 00:50:52.171988] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=16] get_binary_search_range_(ret=0, this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2176854}, scn={val:1736326251986852000, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:52.172483] INFO [COMMON] replace_map (ob_kv_storecache.cpp:776) [46101][TimerWK3][T0][Y0-0000000000000000-0-0] [lt=42] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2025-01-08 00:50:52.174459] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50617][T1001_L5_G0][T1001][YB42C0A89F8B-00062B2D1C8BC36E-0-0] [lt=49][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:52.175762] INFO try_report (ob_tc_stat.cpp:35) [46231][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,124M,0 net_out:1,124M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:3,x,0 t1m0gd:1,x,0 t1m1:3,x,0 t1m1gd:1,x,0 t1m2:3,x,x t1m2gd:1,x,0 t1m0g0:100,x,0 t1m1g0:100,x,0 t1m2g0:100,x,x t1002m0:4,x,0 t1002m0gd:1,x,0 t1002m1:4,x,0 t1002m1gd:1,x,0 t1002m2:4,x,x t1002m2gd:1,x,0 t1002m0g0:100,x,0 t1002m1g0:100,x,0 t1002m2g0:100,x,x t1001m0:4,x,0 t1001m0gd:1,x,0 t1001m1:4,x,0 t1001m1gd:1,x,0 t1001m2:4,x,x t1001m2gd:1,x,0 t1001m0g0:100,x,0 t1001m1g0:100,x,0 t1001m2g0:100,x,x|stat: root:328K/77:110648 disk:328K/77:110648 t1m2:44K/10:60619 t1m2g0:44K/10:60619 t1002m2:200K/49:120536 t1002m2g0:200K/49:120536 t1001m2:84K/18:111524 t1001m2g0:84K/18:111524 |q0: 0 0 0 |q1: 0 0 0 [2025-01-08 00:50:52.175830] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=56] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8562, offset=2428680}, end_cursor:ObLogCursor{file_id=1, log_id=8563, offset=2428977}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946b8050, len:4344, log_data_len:297, seq:8562, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.175972] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326251986852000, v:0}, ls_id={id:1002}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:52.176005] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436803442000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:52.176025] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1002}) [2025-01-08 00:50:52.176054] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:745) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1003}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-01-08 00:50:52.177293] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1430) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=19] succ to get rec scn(*this={this:0x2ad8dc2b6030, ls_id:{id:1003}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1003}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:1736324436825152001, v:0}, max_applied_start_working_ts_:{val:1736324436825152001, v:0}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1736324445275187, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:6}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.177388] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=94] tx ctx memtable get rec scn(this={ObITable:{this:0x2ad8fc5998f0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2ad8fc5998f0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1003}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-01-08 00:50:52.177631] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=219] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326246693924003, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.177695] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:637) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=61] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1003}) [2025-01-08 00:50:52.177707] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:501) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1003}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1736324436825152000, v:0}) [2025-01-08 00:50:52.177720] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:50:52.177734] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1220) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1003}) [2025-01-08 00:50:52.177748] INFO [PALF] get_binary_search_range_ (palf_handle_impl.cpp:2098) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=10] get_binary_search_range_(ret=0, this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, min_block_id=0, max_block_id=0, result_block_id=274894685184, committed_lsn={lsn:2166529}, scn={val:1736326251703671004, v:0}, last_locate_scn={val:18446744073709551615, v:3}, last_locate_block=274894685184) [2025-01-08 00:50:52.179600] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50464][T1002_ArcSender][T1002][YB42C0A89F89-00062B2D1C28458E-0-0] [lt=25][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:52.179633] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [46347][T1_ArcSender][T1][YB42C0A89F89-00062B2D1688458E-0-0] [lt=26][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:52.180317] INFO [STORAGE.REDO] notify_flush (ob_storage_log_writer.cpp:552) [50442][T1002_OB_SLOG][T1002][Y0-0000000000000000-0-0] [lt=22] Successfully flush(log_item={start_cursor:ObLogCursor{file_id=1, log_id=8563, offset=2428977}, end_cursor:ObLogCursor{file_id=1, log_id=8564, offset=2429274}, is_inited:true, is_local:false, buf_size:8192, buf:0x2ad8946c0050, len:4047, log_data_len:297, seq:8563, flush_finish:false, flush_ret:0}) [2025-01-08 00:50:52.180570] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:215) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] update clog checkpoint successfully(clog_checkpoint_lsn={lsn:0}, checkpoint_scn={val:1736326251703671004, v:0}, ls_id={id:1003}, service_type="MAX_DECIDED_SCN") [2025-01-08 00:50:52.180655] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1664) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=78][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1003, self:"192.168.159.137:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1736324436825152000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:0}, lsn={lsn:0}) [2025-01-08 00:50:52.180676] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:190) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:0}, ls->get_ls_id()={id:1003}) [2025-01-08 00:50:52.180686] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:197) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=4) [2025-01-08 00:50:52.199328] INFO [SHARE] update_decay_factor_ (ob_throttle_unit.ipp:284) [50594][T1001_TenantWea][T1001][YB42C0A89F89-00062B2D1CD8AB9E-0-0] [lt=41] [Throttle] Update Throttle Unit Config(is_adaptive_update=true, N=7.718940544128417969e+01, this=0x2ad8fb769208, enable_adaptive_limit_=true, Unit Name=TxShare, Config Specify Resource Limit(MB)=511, Resource Limit(MB)=385, Throttle Trigger(MB)=231, Throttle Percentage=60, Max Duration(us)=7200000000, Decay Factor=7.906445290688600380e+02) [2025-01-08 00:50:52.199964] INFO [LIB] stat (utility.h:1210) [50594][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=162] [PALF STAT APPEND COST TIME](cur_stat_count=19, stat_interval=1000000, avg (count/cost)=0, this=0x2ad908dae050, extra_info="id:1") [2025-01-08 00:50:52.202106] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:254) [50416][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] ====== traversal_flush timer task ====== [2025-01-08 00:50:52.202416] INFO [MDS] flush (mds_table_mgr.cpp:195) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185383-0-0] [lt=36] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:50003}, rec_scn:{val:1736324427835279009, v:0}}, {tablet_id:{id:50114}, rec_scn:{val:1736324427835279012, v:0}}, {tablet_id:{id:50528}, rec_scn:{val:1736325330964868004, v:0}}, }, max_consequent_callbacked_scn={val:1736326251909938002, v:0}, *this={this:0x2ad8fc4a7150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736324403874012002, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324407906905006, v:0}, all_id_meta:{id_meta:[{limited_id:1736326260087570000, latest_log_ts:{val:1736326243478874001, v:0}}, {limited_id:1000001, latest_log_ts:{val:1736324403874012002, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.202619] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185383-0-0] [lt=166] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad91ad41150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251909938001, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436756803999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:1, proposal_id:1, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.202735] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185383-0-0] [lt=84] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad92432d150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251986852000, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436803442001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.202855] INFO [MDS] flush (mds_table_mgr.cpp:172) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185383-0-0] [lt=87] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2ad93a267150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1003}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1736326251703671004, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1736326249485990004, v:0}, tablet_change_checkpoint_scn:{val:1736324436825151999, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:0, palf_env_:0x2ad8f0ff8030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:100, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-01-08 00:50:52.203368] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [50416][T1002_TimerWK1][T1002][YB42C0A89F89-00062B2D1D185383-0-0] [lt=509] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-01-08 00:50:52.204310] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [50459][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-01-08 00:50:52.208213] INFO [STORAGE.TRANS] get_number (ob_id_service.cpp:393) [46099][TimerWK1][T1][Y0-0000000000000000-0-0] [lt=38] get number(ret=0, service_type_=0, range=1, base_id=1736326252204405000, start_id=1736326252204405000, end_id=1736326252204405001) [2025-01-08 00:50:52.211778] INFO [SHARE] delete_tenant_all (ob_disk_usage_table_operator.cpp:231) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=111] delete successful (ret=0, tenant_id=500) [2025-01-08 00:50:52.211899] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46099][TimerWK1][T1][Y0-0000000000000000-0-0] [lt=24] get_stable_disk_usage(ret=0, capacity(MB):=2048, used(MB):=144) [2025-01-08 00:50:52.213514] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=15] get_stable_disk_usage(ret=0, capacity(MB):=512, used(MB):=105) [2025-01-08 00:50:52.214895] INFO [PALF] get_stable_disk_usage (palf_env_impl.cpp:934) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=13] get_stable_disk_usage(ret=0, capacity(MB):=4608, used(MB):=279) [2025-01-08 00:50:52.215024] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:586) [50461][T1002_LSArchive][T1002][YB42C0A89F89-00062B2D1BE8458E-0-0] [lt=22] gc stale ls task succ [2025-01-08 00:50:52.216414] INFO [STORAGE] get_tmp_file_disk_usage (ob_sn_tmp_file_manager.cpp:213) [46099][TimerWK1][T1][Y0-0000000000000000-0-0] [lt=13] get tmp file macro block count(ret=0, ret="OB_SUCCESS", used_page_num=0, macro_block_count=0) [2025-01-08 00:50:52.216431] INFO [STORAGE] get_tmp_file_disk_usage (ob_sn_tmp_file_manager.cpp:213) [46099][TimerWK1][T1001][Y0-0000000000000000-0-0] [lt=15] get tmp file macro block count(ret=0, ret="OB_SUCCESS", used_page_num=0, macro_block_count=0) [2025-01-08 00:50:52.216441] INFO [STORAGE] get_tmp_file_disk_usage (ob_sn_tmp_file_manager.cpp:213) [46099][TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=7] get tmp file macro block count(ret=0, ret="OB_SUCCESS", used_page_num=0, macro_block_count=0) [2025-01-08 00:50:52.218033] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=7][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963659a18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27791} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae19970 trans_version={val:1736322640977792000, v:0} scn={val:1736322640977792000, v:0} tx_id={txid:1449} prev=0x2ad8dae197b8 next=(nil) modify_count=1 acc_checksum=460374033 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:7} write_epoch=3, snapshot_version={val:1736326252216686000, v:0}, *this={this=0x2ad8dae19658 latch_=locked flag=11 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x2ad8dae19970 latest_compact_node=(nil) max_trans_version={val:1736326191777235000, v:0} max_trans_id=27369 max_elr_trans_version={val:1736326191777235000, v:0} max_elr_trans_id=27369 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1736322640977792000, v:0} min_modify_scn={val:1736322640977792000, v:0}}) [2025-01-08 00:50:52.218082] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=48][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963659a18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27791} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:92} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282117672, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282117672, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252216686000, v:0}, tx_id:{txid:27791}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27791}, tx_desc:{this:0x2ad8bcd27fa0, tx_id:{txid:27791}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252216686, active_ts:1736326252217943, commit_ts:-1, finish_ts:-1, timeout_us:29999986, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252315173, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.218168] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=85][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27791}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.218185] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=16][errcode=0] primary key duplication conflict(key={"BIGINT":1},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant file data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27791} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.218214] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=27][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.218316] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=96][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c3b78540, local_datums:0x2ad8c3b78540}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783cc, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c3b783d4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783e3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b783eb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x2ad8c3b783f3, hex: 74656E616E742066696C652064617461, cstr: tenant file data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78403, hex: 6E417AF52D2B0600, int: 1736326252216686}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7840b, hex: 6E417AF52D2B0600, int: 1736326252216686}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b78413, hex: 72D77F0100000000, int: 25155442}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c3b7841b, hex: 00F0120200000000, int: 34795520}]}) [2025-01-08 00:50:52.218435] INFO [STORAGE] print_count (ob_query_iterator_factory.cpp:50) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=76] ObQueryIteratorFactory statistics(multi_scan_merge_alloc_count=0, multi_scan_merge_release_count=0, multi_get_merge_alloc_count=0, multi_get_merge_release_count=0, table_scan_alloc_count=0, table_scan_release_count=0, insert_dup_alloc_count=5510, insert_dup_release_count=5510, col_map_alloc_count=0, col_map_release_count=0) [2025-01-08 00:50:52.218467] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=18][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.218525] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285243-0-0] [lt=9] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27791} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.222827] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:529) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4152] get int failed(name="finish_data_version", ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:52.222852] WDIAG [SHARE] get_finish_data_version (ob_global_stat_proxy.cpp:296) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4152] get failed(ret=-4152, ret="OB_ERR_NULL_VALUE") [2025-01-08 00:50:52.227393] WDIAG [SQL.ENG] get_next_left_row (ob_nested_loop_join_vec_op.cpp:167) [50384][T1002_TenantInf][T1001][YB42C0A89F89-00062B2D1DB87019-0-0] [lt=12][errcode=-4008] failed to get next left row from friver iterator(ret=-4008) [2025-01-08 00:50:52.227459] WDIAG [SERVER] get_int (ob_inner_sql_result.cpp:502) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4152] get obj error(ret=-4152) [2025-01-08 00:50:52.227474] WDIAG [SQL] build_service_name_ (ob_service_name_proxy.cpp:586) [50384][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4152] fail to get column in row. (column_name="service_name_id", ret=-4152) [2025-01-08 00:50:52.227877] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] insert successful (ret=0, tenant_id=1, file_type=0, data_size=25155442) [2025-01-08 00:50:52.229312] WDIAG [STORAGE.TRANS] lock_row_on_frozen_stores_ (ob_memtable.cpp:1335) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285246-0-0] [lt=26][errcode=-5024] internal lock row failed(ret=-5024, ret="OB_ERR_PRIMARY_KEY_DUPLICATE") [2025-01-08 00:50:52.229388] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285246-0-0] [lt=73][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.229402] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285246-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c21e0510, local_datums:0x2ad8c21e0510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e039c, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c21e03a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 18, flag: 0, null: 0, ptr: 0x2ad8c21e03c3, hex: 74656E616E74206261636B75702064617461, cstr: tenant backup data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03d5, hex: 206D7AF52D2B0600, int: 1736326252227872}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03dd, hex: 206D7AF52D2B0600, int: 1736326252227872}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03e5, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c21e03ed, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:50:52.229512] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285246-0-0] [lt=56][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.229558] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285246-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27792} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.230462] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:257) [50598][T1001_ArcSender][T1001][YB42C0A89F89-00062B2D1CE8458E-0-0] [lt=27][errcode=-4018] try free send task failed(ret=-4018) [2025-01-08 00:50:52.232686] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=27] insert successful (ret=0, tenant_id=1, file_type=8, data_size=0) [2025-01-08 00:50:52.234125] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=52][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad963659f18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27793} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3d9a8 trans_version={val:1736324383117027001, v:0} scn={val:1736324383117027001, v:0} tx_id={txid:14338} prev=(nil) next=(nil) modify_count=0 acc_checksum=1983279813 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326252232880000, v:0}, *this={this=0x2ad8dae3d918 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3d9a8 latest_compact_node=(nil) max_trans_version={val:1736326191821630001, v:0} max_trans_id=27371 max_elr_trans_version={val:1736326191821630001, v:0} max_elr_trans_id=27371 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383117027001, v:0} min_modify_scn={val:1736324383117027001, v:0}}) [2025-01-08 00:50:52.234262] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=133][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad963659f18 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27793} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282133946, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282133946, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252232880000, v:0}, tx_id:{txid:27793}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27793}, tx_desc:{this:0x2ad8bcd36240, tx_id:{txid:27793}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252232880, active_ts:1736326252234059, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252335342, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.234351] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=89][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27793}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.234367] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=15][errcode=0] primary key duplication conflict(key={"BIGINT":1001},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27793} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.234454] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=85][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.234465] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=8][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:16, datums:0x2ad8c1d1a510, local_datums:0x2ad8c1d1a510}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a39c, hex: E903000000000000, int: 1001}, col_id=1:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a3a4, hex: 3139322E3136382E3135392E313337, cstr: 192.168.159.137}, col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3b3, hex: 420B000000000000, int: 2882}, col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3bb, hex: 0100000000000000, int: 1, num_digit0: 0}, col_id=4:{len: 15, flag: 0, null: 0, ptr: 0x2ad8c1d1a3c3, hex: 74656E616E7420746D702064617461, cstr: tenant tmp data}, col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3d2, hex: B0807AF52D2B0600, int: 1736326252232880}, col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3da, hex: B0807AF52D2B0600, int: 1736326252232880}, col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3e2, hex: 0000000000000000, int: 0}, col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x2ad8c1d1a3ea, hex: 0000000000000000, int: 0}]}) [2025-01-08 00:50:52.234584] WDIAG [STORAGE] insert_row (ob_ls_tablet_service.cpp:3334) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=89][errcode=-5024] failed to write row(ret=-5024) [2025-01-08 00:50:52.234621] INFO [STORAGE.TRANS] rollback (ob_memtable_context.cpp:853) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D17285249-0-0] [lt=12] memtable handle rollback to successfuly(from_seq_no={branch:0, seq:5}, to_seq_no={branch:0, seq:3}, remove_cnt=1, elapsed=0, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=0 trans_id={txid:27793} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=0 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=1, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(-1,0,0,0,1,1,1,0)|]}}) [2025-01-08 00:50:52.237368] INFO [SHARE] update_tenant_space_usage (ob_disk_usage_table_operator.cpp:126) [46099][TimerWK1][T0][Y0-0000000000000000-0-0] [lt=23] insert successful (ret=0, tenant_id=1001, file_type=3, data_size=0) [2025-01-08 00:50:52.238614] WDIAG [STORAGE.TRANS] mvcc_sanity_check_ (ob_mvcc_row.cpp:956) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=24][errcode=-5024] find double insert node(ret=-5024, node=this=0x2ad96365a190 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27794} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, prev=this=0x2ad8dae3db90 trans_version={val:1736324383129258001, v:0} scn={val:1736324383129258001, v:0} tx_id={txid:14339} prev=(nil) next=(nil) modify_count=0 acc_checksum=450727420 version=1736322640966574 type=0 flag=4 snapshot_barrier=1736326252067809000 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, snapshot_version={val:1736326252238381000, v:0}, *this={this=0x2ad8dae3db00 latch_=locked flag=11 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x2ad8dae3db90 latest_compact_node=(nil) max_trans_version={val:1736326191826572001, v:0} max_trans_id=27372 max_elr_trans_version={val:1736326191826572001, v:0} max_elr_trans_id=27372 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1736324383129258001, v:0} min_modify_scn={val:1736324383129258001, v:0}}) [2025-01-08 00:50:52.238654] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_row.cpp:1018) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=40][errcode=-5024] mvcc write failed(ret=-5024, node=this=0x2ad96365a190 trans_version={val:4611686018427387903, v:0} scn={val:4611686018427387903, v:0} tx_id={txid:27794} prev=(nil) next=(nil) modify_count=4294967295 acc_checksum=0 version=1736322640966574 type=0 flag=128 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:86} seq_no={branch:0, seq:4} write_epoch=1, ctx={this:0x2ad88514a488, ls_id:{id:1}, ls:0x2ad8aec7a150, branch:0, timeout:1736326282138395, tablet_id:{id:309}, table_iter:0x2ad885148978, table_version:1736322555877896, mvcc_acc_ctx:{type:3, abs_lock_timeout_ts:1736326282138395, tx_lock_timeout_us:-1, snapshot:{version:{val:1736326252238381000, v:0}, tx_id:{txid:27794}, scn:{branch:0, seq:2}}, tx_table_guards:{tx_table_guard:{tx_table:0x2ad8aec85b10, epoch:1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:[0:{}, 1:{}, 2:{}, 3:{}]}, src_ls_handle:{ls_map_:null, ls_:null, mod_:-1}}, tx_id:{txid:27794}, tx_desc:{this:0x2ad8bcd341e0, tx_id:{txid:27794}, state:1, addr:"192.168.159.137:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1736326252238381, active_ts:1736326252238381, commit_ts:-1, finish_ts:-1, timeout_us:29999989, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1736322126, cluster_version:17180067072, seq_base:1736326252340141, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, conflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}, tx_ctx:0x2ad89c082950, mem_ctx:0x2ad89c083ed0, tx_scn:{branch:0, seq:4}, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:1, is_write_only_index:0, is_check_row_locked:0, is_lob_aux:0, is_skip_flush_redo:0, is_update_uk:0, is_update_pk_dop:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, is_read_store_ctx:false, update_full_column:false}) [2025-01-08 00:50:52.238724] WDIAG [STORAGE.TRANS] mvcc_write (ob_mvcc_engine.cpp:337) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=68][errcode=-5024] mvcc write failed(ret=-5024, arg={tx_id:{txid:27794}, data:0x2ad885147348, old_row:null, modify_count:4294967295, acc_checksum:0, memstore_version:1736322640966574, seq_no:{branch:0, seq:4}, write_epoch:1, scn:{val:4611686018427387903, v:0}, column_cnt:9}) [2025-01-08 00:50:52.238738] WDIAG [STORAGE.TRANS] on_key_duplication_retry (ob_memtable_context.cpp:329) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=13][errcode=0] primary key duplication conflict(key={"BIGINT":1002},{"VARCHAR":"192.168.159.137", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant tmp data", collation:"utf8mb4_general_ci"}, this={ObIMvccCtx={alloc_type=0 ctx_descriptor=0 min_table_version=1736322555877896 max_table_version=1736322555877896 trans_version={val:4611686018427387903, v:0} commit_version={val:0, v:0} lock_wait_start_ts=0 replay_compact_version={val:0, v:0}} end_code=0 tx_status=0 is_readonly=false ref=1 trans_id={txid:27794} ls_id=1 row_callback[alloc:0, free:0, unsubmit:0] redo[fill:0,sync_succ:0, sync_fail:0] main_list_len=1 pending_log_size=0 callback_list:{cnt=1 stat:[tx_end=0, rollback_to=0, fast_commit=0, remove_memtable=0, ext_info_log=0] detail:[(log_epoch,length,logged,synced,appended,removed,unlog_removed,branch_removed)|0:(1,1,0,0,1,0,0,0)|]}}) [2025-01-08 00:50:52.238761] WDIAG [STORAGE] insert_row_without_rowkey_check (ob_tablet.cpp:4652) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=22][errcode=-5024] fail to set memtable(ret=-5024) [2025-01-08 00:50:52.238771] WDIAG [STORAGE] insert_row_to_tablet (ob_ls_tablet_service.cpp:5675) [46099][TimerWK1][T1][YB42C0A89F89-00062B2D1728524C-0-0] [lt=6][errcode=-5024] failed to write table row(ret=-5024, table id=309, col_descs=[cnt:9, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], datum_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, co